It's been a while since the last release! But we're happy to announce that RhodeCode 4.17 (along with 4.16) is officially out!
Those releases highlights are:
- New enhanced User Interface.
- ElasticSearch 6.X support and new enhanced full-text search.
- Artifacts feature to store binary/none-vcs files bound to a repository.
- Authentication: introduced User restriction and Scope restrictions.
- SVN PreCommit hooks, allowing file name and size validation server side.
- Mercurial 4.9 Support with sparserevlog.
Take a few minutes to update your RhodeCode instance:
rccontrol self-update && rccontrol upgrade '*'
New to RhodeCode? Download the latest RhodeCode 4.17 Series from our website.
Keep reading for full release details.
New UI
RhodeCode UI wasn't updated for a longer period, we knew it's getting dated and we needed a progressive enhancement.
We're dedicating this and the next few releases to enhance our UI and UX.
The first iteration of changes is shipped together with 4.17 release. We worked on some obvious UX problems and based on that
we started to change UI to make it clear, less cluttered and nicer looking.
We're still not there yet, but since we want to ship as often as possible we'll keep on working on the UX/UI in the next few releases.
A big chunk of that pending work is code-review / pull-requests which are the most complex.
There's a lot of feedback gathered over time from our team, and our users! We want to take some time and produce best possible outcome.
We recommend taking a look at https://code.rhodecode.com to explore the new UI and enhancements we made.
Enhanced Full-Text Search
Full-text search has been a feature in RhodeCode for a very long time. Initially with the ability to do a basic search within all projects, and didn't evolve past that for several previous releases.
We knew we could do much better and took our time on what the next search capabilities could look like.
In 4.17 release we're introducing support for elastic search 6.X new query language, and new much more efficient indexer.
Let's go briefly through the changes we shipped in 4.16 and 4.17 releases.
Enhanced Full-Text Search - new indexer
Previously the indexer had some troubles with really big data sets. It would require a lot of resources to perform indexing often running out of memory.
We knew we need to improve how actually we feed the data to the full-text-search index.
The indexer has been rebuild from scratch to accommodate with repositories that have 100s of GB of data and commit history spawning more than 500 000 commits. It got few extra flags to be able to
rebuild index simply for single repositories, or groups of repositories. There's also a new flag to purge some data from index easily.
We're happy with the result and we're sure for our customers it's going to be much easier to build index very often for their big repositories.
Enhanced Full-Text Search - new query syntax
In previous releases, we used a single subset of query language that was shared between Whoosh (CE search backend) and ElasticSearch (EE search backend)
This gave some benefits in terms of testing, and ease of switching between Those two. We had some new requirements from full-text search and we quickly realized that current query language wouldn't be able to fulfill those.
Starting from 4.17 ElastiCSearch backend uses the new query language from Lucene. This solves one important requirement that we had, ie. write advanced search queries to get better results.
With the new query lang it's possible to write such nice queries like those examples below:
# find "security" in commit messages in two repositories security repo_name:(rhodecode-vcsserver OR rhodecode-enterprise-ce) # search for TODO string in .conf files for all repositories that start with 'develop-' repo_name:develop-* file:*.conf TODO # knowing that some files have a lot of TODO words automatically generate, you can exluce them easily ! repo_name:develop-* file:*.conf -file.raw:rhodecode\/lib\/dbmigrate\/schema\/*.py -file.raw:rhodecode\/public\/js\/rhodecode\/i18n\/*.js TODO # maybe you would like to know how many big javascript files your project have? Sure! now very easy # search for .js files in all your repositories that are over 200kb in size extension:js size:[204800 TO *]
UX with search
We did also many changes in how we show results. It's very easy now to narrow the scope to search within only a single repository group, or single repository from global search results.
The match highlighter is also improved and it's using the native implementation from ElasticSearch to produce very easy to browse results.
Furthermore, the top search field is now context-aware, if you're in a repository commits page, it would prioritize the results from commit message of that repository, same for files view.
Repository group view prioritizes the search inside that group and so on
Full-Text Search enahancements summary
By default and for legacy reasons new capabilities for search aren't enabled.
To use new features a full index rebuild is required, in addition --es-version=6 flag needs to be used with indexer and search.es_version = 6 should be set in rhodecode.ini
Additionally, new mapping format is available for the indexer that has additional capabilities for include/exclude rules.
The old format should work as well, but we encourage to generate a new mapping.ini file using rhodecode-index command, and migrate your repositories to the new format.
We're very happy about the end result, and we believe RhodeCode now has one of the best full-text search capabilities in this space!
Repository Artifacts (BETA)
We're happy to present a brand new feature called Artifacts to help our users leverage RhodeCode system for advanced devops usage.
Artifacts feature is a binary file store that is wrapped around our best-in-class permission system.
We took a look into some leading solutions on the market (such as jFrog Artifactory) and found out that adding a similar solution to RhodeCode system would benefit our users.
Since storing binary files in DVCS systems is usually problematic, dedicated binary storage exposes many advantages.
We imagine typical use cases for it would be to let the CI/CD systems store generated build binaries and store it next to the repository context. We Added web-interface and APIs
for Artifacts so external accounts such as CI-Bot can upload artifacts right there back into RhodeCode. RhodeCode can then serve as a reliable, fast, highly-scalable binary file server.
It allso allows users inside the company to share larger binary files without using DVCS system and it's history.
There are many more use-cases for Artifacts, for example, we internally
use it to share CI build logs on failed commits. This solves access problems to the CI for external users.
Our Jenkins CI generates new artifacts with build logs and exposes them to users within RhodeCode systems, so we don't need to share CI access for developers to gain information about detailed build failures.
Artifacts feature is wrapped in our well know permission system, so it's also easy to control Artifacts access by controlling repository access.
Repository Artifacts is in BETA, we'll going to gather all the user feedback and also our internal feedback to polish that feature and make it out of beta for 4.18 release.
Authentication scopes
We introduced User restriction and Scope restriction for RhodeCode authentication plugins.
Instance administrators can now limit the usage of RhodeCode plugins to super-admins user types, and usage in Web, or VCS protocol only.
This is mostly to help to migrate users to SAML, keeping the super-admins to manage instances via local-logins,
and secondly to force usage of AuthenticationTokens instead of re-using the same credentials for WEB and VCS authentication.
Security Improvements
Security is always a top priority here at RhodeCode.
In this release we fixed the following important issues:
- Fork page: don’t expose the fork origin link if we don’t have permission to access this repository.
- Security: fix possible XSS in the issue tracker URL.
- Security: sanitize plaintext renderer with bleach, preventing XSS in rendered HTML.
- Audit logs: added audit logs for API permission calls.
- SSH: replaced pycrypto with cryptography to generate SSH keys as pycrypto isn’t considered safe anymore.
If you use any of the above, we highly recommend upgrading to the latest version.
Next Release
Starting from this blog post we're going also to highlight what we're working on and what to expect from new releases.
For upcoming 4.18 release, we're working on improvements on memory usage and performance.
VCSServer will be getting a whole new caching layer that would greatly benefit from distributed caches such as Redis.
We also re-wrote the GIT backend to use libgit2 for GIT which brings nice improvements on bigger repositories.
Summary
See the full list of changes in the 4.16 release release notes.
See the full list of changes in the 4.17 release release notes.
As usual, the update process is a simple one-liner type:
rccontrol self-update && rccontrol upgrade '*'
in the command line interface. Done!
Sincerely yours,
The RhodeCode team.