Allow users to install websites from version control

Jacob Brown shared this idea 1 year ago
In Progress

Just a small feature request we think would be useful for clients utilizing web files that get version controlled through svn, git or mercurial.

Our proposed implementation would be to have a VC control tools (ie a GIT button) in the File Manager that allows users to pull/push/checkout/checkin repositories to the directory they choose, and keep it updated.

The current method is to download the zip from git and upload it via the file manager, over-writing files as it goes however we propose the VC method to allow users to roll back after a failed deployment.

Best Answer
photo

For those interested in installing a "not even edge yet" build and playing with this feature, please hit me up! adam.french@cpanel.net - I've got to send you install instructions!

Comments (52)

photo
2

Very interesting. How would you expect database migrations (upgrades to the schema, etc) or cache flushes to be handled?

photo
1

Very good point. In terms of CI integration and the like, I'm not totally sure however I imagine that if you're utilizing something some sort of automation, you wouldn't be working with cPanel. Especially now that people are moving towards Docker and it's friends.

Majority of scripts do release an "update" script which handles database alterations. Most scripts I've worked with also don't VC config files, and handle updates to the config file through a script designed to non-destructively change it. (ie wp-config.php is wp-config.php.sample)

photo
1

We could look for a Makefile and run make if we find it. Developers could lump in whatever their needs are into the default make task and dispatch the required tasks from there.

photo
1

Symfony/doctrine and Laravel use the so called migrations that are updated or rolled back through a console command.

Another command also rebuilds the composer autoload cache.

Perhaps along with the remote URL in each site the user could define a couple of commands to be executed after the synchronization.

Alternatively, something like the travis yml could be loaded in the remote rep, to define the commands that need to be executed.

photo
photo
3

This is a feature that I personally feel is extremely valuable to our more technical users, as this is how modern developers interact with code and push code around. When looking the ability to deploy from version control I can see quite a few advantages and disadvantages. When discussing this feature I will use 'git' however that does not mean we are unwilling to look into mercurial or svn support.

I do feel that just building this into the file manager, while possibly a great minimum viable product for this feature, is not harnessing the entire power of git for this feature.

Below are some of the various use cases that I have come up with while grokking this feature:

  • Ability to clone a 3rd party software repository and deploy/update a web app via git
  • Ability to maintain one's own web content from a 3rd party hosted git repository
  • Ability for the host to maintain a list of allowed git repositories that users can deploy software from.

The usual feedback I hear on this is that this is too technical of a feature for use by your average cPanel user.

photo
2

It is definitely how most devs push code around. It's how we've done it at all my jobs, it's how I do it for my personal projects and it's how my friends prefer to share code with each other.

Ideally, you would be able to:

  • Clone
  • Commit/add
  • Push
  • Pull
  • Revert
  • Switch branch

I like the idea of whitelisting/blacklisting repositories. Whether this should be a WHM or cPanel managed list, I'm not sure of. That might be getting a bit too crazy, but it would be good for companies (like the one I work for) who utilize cPanel for "microsites or blogs", but want to maintain that extra security where people can only download from our repos.

It is definitely too technical for the lower level cPanel users (guys who pay $30/month for 2GB of webspace because the hosting company will do most changes to their account, like creating emails, databases, subdomains, etc), but for the mid-level and higher-level cPanel users (casual devs, guys who aren't blase about computers, and professional devs), it would be a welcomed featured.

Most low level guys will just ignore it anyway :P

photo
photo
2

I would better encourage some version controlling system (both remote and on server) so any changes under specific repository over a specific branch can be used for this.

I personally don't feel the cPanel will be interested as this feature involves moderator controls and understanding however for cPanel, I would love to see this as simple as it can i.e. some options where user can change his mode to developer mode and than all such settings may start appearing.

Anyways, I am strongly supporting this feature as in many cases with me, I have to alter system configurations and have to provide shell access to my clients requiring this feature with cpanel which makes the system unstable farmost time.

photo
1

Version control from the shell can already be accomplished. After reading this I decided to check and can confirm that I can checkout files from my SVN repo using a "normal shell" and "jailed shell" using a cPanel accounts login.

What I would really like to see is it implemented into the file manager so that SSH isn't required.

Going by Adams response, I feel that they might be interested and I also feel that it would be a nice additional feature for their hosts to be able to boast about.

photo
photo
1

Honestly, this feature is not necessary. There are more important features that cPanel needs to focus their time on than something the "more technical users" as one puts it, can simply setup themselves. SNV is just about obsolete, and who uses Mercurial? Git is what most everyone uses, and the ones that don't, should.

BUT... What would be nice is if cPanel installed Git by default with itself, like it does everything else, that way developers working on cPanel servers that don't have root access can setup Git repos per project/user.

IF... you do have root access, it is fairly simple to setup Git on cPanel servers. I currently am and have been using Git for my projects for some time now on cPanel servers with no issues. The setup isn't much different, just requires a few extra steps to get it working the "cPanel" way.

As Root:

1. Verify that the /etc/yum.conf file does not contain perl* in the exclude line.

2. yum install git

3. nano /opt/suphp/etc/suphp.conf

change: allow_file_group_writeable=true

change: allow_directory_group_writeable=true

As User: (not root, or you will run into all kinds of permission errors)

1. Enable shell access for user account - Jailed Shell or Normal Shell (Doesn't matter, they both work)

2. Login As User, and setup your repo...

I personally like to initialize my repo before the public_html directory, and follow the cPanel structure in my repo for better compatibility with cPanel severs. But, everyone has their own weird way of doing things, so do whatever makes the most sense to you.

My Way example:

SSH to /home/some-user-account

Type: git init

Type: git config --global user.name "FirstName LastName"

Type: git config --global user.email "youremail.com"

Type: git config --global core.excludesfile ~/.gitignore

Type: git remote add origin https://somerepo.git

/home/some-user-account/.gitignore: (ignore everything but public_html directory)

  1. /*
  2. !/public_html

This .gitignore file is basically for cpanel stuff, you can add another to the public_html directory to ignore project files. Git supports multiple .gitignore files and can be placed just about anywhere. They work the same exact way robots.txt files do. So if you want Git to ignore for example, your cache folders and all xml files, just create another .gitignore file and place it inside your public_html directory.

  1. *.xml
  2. cache/

Anytime you want to deploy, just login as user and type: git pull origin master (or whatever branch u want to pull).

photo
1

As I stated earlier (but after you posted this), you can do it from a shell provided that the web host sets it up on their server. But are webhosts going to invest the time and money into securely setting it up on all their nodes for their clients? Are they going to want to invest in that risk?

Larger ones; sure. Mid-tier and smaller companies; less likely.

Adding this feature would just be yet a nicer thing for cPanel to provide to their websites. At no point have I stated it's necessary, I just said that me - and some other devs I work with - would think it would be nice.

photo
1

I agree that it would be nice. Why I voted thumbs up, even though I highly doubt cPanel will invest the time to implement it.

photo
photo
2

While highly technical in nature, this idea has potential for end-users if it's executed properly. The idea of designers providing a GIT repo URL to customers...and all the customer has to do is register it within cPanel and hit deploy...that has sexy written all over it.

photo
1

It's something cPanel could roll out in multiple stages over time. A good start would be to just have Git installed at server level by default, so its available at the user level without having to have the server admin install/setup Git. This quick-fix solution would still require the admin to enable shell access for users, but at least Git would already be available to use once a shell is enabled.

Ideally and interface of some sort would be preferred, especially for the non-tech users or in cases where a server admin refuses to enable shell access. Again, this part of it could be rolled out over time. At least the quick-fix solution I suggested would solve the problem for devs that currently have shell access, but not root level access.

I know someone suggested building it into File Manager, but I personally think it would make more sense to build a separate Interface for this, e.g. like Cron Jobs, etc. More simple to maintain and scale since you would essentially be building a stand-alone Git client out of JavaScript.

photo
1

@bruce - your thinking is right in line with how we'd approach this project. As of right now git is already available to end-user cPanel accounts if they have shell access. Beyond that, though, I'm definitely thinking a dedicated interface for configuring git repositories at a file location. (Links to this interface could totally exist inside File Manager so that you could select a folder and start from there...but the feature itself would be outside of File Manager).

photo
photo
1

I'm really curious about this feature. Some of my customers have been asking for this as they are mostly designers and do not have a lot of experience using SSH. Any comments on the status?

photo
1

We haven't considered adding this to the roadmap since I started in this position in March, but I can definitely bring it up again. With only 17 votes over nearly a year, it doesn't seem like this has much interest in the community.

photo
2

I have to use cPanel for clients on a regular basis and I find this extremely frustrating, so much so that I ended up using Plesk instead which has GIT control baked in and can be set up in about 3 clicks. Now I just use subdomains with protected directories that are all set up to automatically pull any changes from GitHub and everything works seamlessly - clients can see me update things in real time.

To confirm cPanelAdamF's statement: It is absolutely sexy.

photo
1

Staging / Development / Production sites with GIT through cPanel. That'd be awesome :)

photo
1

My team and I just had a serious discussion about this with intent to start planning it out. I'll have more to say after we explore the idea more...including a call for focus group members...when we get done with that exploration.

We're excited!

photo
1

Good news!!

A nice feature I like in other services is the ability to configure a webhook (for example at github) so that a pull is triggered whenever a new version is pushed on the master branch.

photo
2

yup, we're looking at that as the main way to be notified of new changes that need to be pulled in. We've also considered a scheduled pull daily (probably during the day rather than overnight).

As far as development goes, first thing is to make the first clones work. Then we'd tackle an 'update now' button...then start talking about triggered updates or regularly scheduled updates or both.

photo
1

Cool :)

Selecting which branch to probe will also be very useful for test/staging environments.The "overnight" thing is a bit dangerous.

What is overnight for one site might be working hours for another.

I would make the execution time configurable per case.

BTW, is this going to be a WHM or Cpanel feature?

Is composer included in installations?

photo
2

Right now our thinking is that this should be a cPanel feature at first. Keeping a directory up to date could be useful at the WHM layer, but we'd like to keep it simple at first.

re working hours - yeah, definitely. We'd rather updates to the site happen automatically & conveniently...and not cause panic at 1am during upcp.

photo
2

Something for your checklist if you are still gathering specs.

It would be nice to be able to configure pre and post sync scripts. That way things like cache clear, composer installations or updates and the like can be automated too.

Also, it would be nice to have a way to define environment variables and .env files (used by laravel and symfony), although this can be probably handled manually from the file manager.

photo
2

On it. We were already thinking about pre-sync/post-sync scripts. Hadn't thought of ENV variables, though, glad you said something about that!

photo
1

@dimkasta - Something to keep in mind is that the work for https://features.cpanel.net/topic/ruby-app-support-through-phusion-passenger will already have the ability to specify ENV variables for the launch of processes. Are you wanting additional set of ENV vars for the pre-flight and post-flight scripts too?

photo
1

Thanks for the tip

I will check it, although I do not use ruby

In any case, the file manager works ok for those files

photo
photo
2

This is now officially planned for v68, baring unforeseen circumstances!

photo
1

Nice!

photo
photo
1

A lot of this is already do-able in v64 and newer. We're prepping some documentation for advanced users comfortable at the command line that will describe the way to achieve it right now.

Don't worry. Still building out a nice new feature in the cpanel interface for it too ;-)

photo
1

New documentation for how to do it right now (v64 or newer) is now available.

https://documentation.cpanel.net/display/CKB/How+to+Host+Git+Repositories+on+a+cPanel+Account

As a result of working up this doc, the team and I identified three major scenarios that we need to plan for in the development of this feature:

  1. "Single Remote without automatic deployment" - basically just repository hosting. For people who don't want to use Github or one of the other special purpose hosts.
  2. "Single Remote with automatic deployment" - hosting a repo on your cpanel account with the expectation that new changes are automatically checked-out upon push
  3. "Multiple Remote with automatic deployment" - hosting a clone of a repository hosted elsewhere (eg Github) and automatically monitoring that remote for new updates.

I've got those roughly ordered in terms of complexity. 1 and 2 are already possible right now, and the new document explains how to do it via command-line. 3 is where the fun is ;-)

photo
1

Question for the room: How do you plan on marketing this feature? What would advertisements on your hosting company's website look like for this?

I ask because this will help guide us in how best to chop up this feature's capabilities and how best to schedule out which parts of the project we do first.

photo
4

"Dev like it's 2017, not '97" — perhaps? :p

photo
1

Laravel forge and envoyer might give you nice ideas

photo
1

What would your hosting Packages (like, WHM Packages) look like? Do we need to slice-up the feature into more than one 'feature manager'-entry so that you can fine-tune your offerings?

photo
photo
1

Dev update for everyone:

We've broken ground on the interfaces to create and list git repositories. I'll have some early screenshots to share here pretty soon.

We did identify a couple of things now that we're getting started.

First, we know people will create git repositories from the shell since they have that ability so we're going to implement a 'discover repositories' button which will trigger a scan of the user's home directory for .git folders and update the list of repositories in the interface. (Pushing it more than once in rapid succession will only trigger one scan for that user at one time. Scanning file structures is slow and expensive so we're building it in such a way that it can't really be abused.)

Second, we've discussed performing a 'git stash --include-untracked' before receiving any git-push in order to clear out any local changes that might exist on the cpanel-remote. In the occasion where we end up git-stashing something, we'll shoot a note to the end-user saying "hey, we cleaned up something in order to perform your deployment. Here's the git-stash-apply command to restore it". What do you all think of this idea? Should we simply fail the deployment instead?

photo
2

Adam, someone that uses the shell, probably doesn't use cPanel and I suspect most don't even give shell access for security reasons on shared accounts. The idea and concept of cPanel is having a web GUI for site/server tasks, click and point and as easy as possible for non-computer persons. If this is the initial concept, great, but users will expect to do this from a visual interface as well eventually.

photo
1

The stash idea nice is in cases when someone makes an experimental change on the envronment and does not want to lose it

About the scanning, it really depends on how you are designing the interface. You could just add a folder browser and let the user detect for a git repository or create a new one just on the selected/working folder

photo
1

Stash is indeed good and should be there for sure, other wise any local changes would break the repo pull.

However, I think an auto pull would be great to have in place as well.

For instance on Github, BitBucket and Gitlab is possible to setup a hook when the repo gets updated, in that case a HTTP request will be fired to the given endpoint, the service behind the endpoint can be execute the necessary actions, in this case pulling the repo.

So I guess the implementation of this on cPanel would be, having an endpoint/webhook for each repo so it can pull the repo once it has been accessed.

In this way anytime we push new changes to git repository on github/bitbucket/gitllab, the repo on the cPanel will be updated automatically as well.

The flow is same as any CI is using to run the tests against the code changes, Heroku uses the same flow as well to keep the apps updated with the latest changes on app git repositories.

photo
1

Re shell access - Yup, we understand that too. We're not intending to require shell-access in order to set up a deployment...but we have to account for use-cases where they DO have it and they meddle (because users will meddle in our experience). Plus all of this is happening in user's home-directories so there's potential for meddling via the File Manager as well.

The first versions of this feature will require shell access due simply to the fact that it makes developing it a bit easier on our end. Later versions will remove that requirement and do as much on behalf of a no-shell user as appropriate.

Re webhooks - Already planning to provide webhook URLs for repositories which enable deployment! We're intending to offer them on all repositories that enable deployment. If you don't want to use them in to integrate deployments, you can still bookmark them and they'll work all the same.

As a behind the scenes, here's a chart we came up with that shows what's going to happen during a deployment and when it'll happen.

photo
1

Another feature that would push this to an entirely new level, is to allow the user to change the working root public directory.

That way, we can create folders to host different repos, versions and/or environments, make and test all building tasks and easily switch between them by pointing the site to some other directory.

This can make deployments, migrations, cache clears, cache warmups and total rollbacks much faster and allow us to minimize downtime.

This can already be done by using subdomains, but we cannot point the root domain to some other directory to make environment switches completely effortless and allow for instant rollbacks that allow you to troubleshoot the new environment right at the prod server

photo
2

@dimkasta - I hesitate to stray into the management of the root domain's docroot for this particular project (because other teams are in that arena...plus it's a major systemic change that is going to be rough to do for unrelated reasons) HOWEVER we can certainly prepare docs for how to do the "docroot-swap" dance with the git deployment feature for addon domains or subdomains.

I'd like to get this feature out the door and used in the real world some before we go about making the 'docroot swap' easier to do...but that's not to say no to the idea. Only 'not yet'. We're currently trying to design this in such a way that it's agnostic to whether your repository is in a docroot or not. (Meaning it'll act the same whether you're deploying a website or a set of files and relying on Indexes to list them OR whether you just want to use git over SSH or HTTPS). Once we get the basics down and implemented, we can refine it further.

photo
1

Yep, perfectly understandable approach.

Just for the record and to give you a few ideas, two deployment systems that I am currently experimenting with, are both using a symlink for the docroot.

You have a "releases" folder where you can clone your repo in folders named 1, 2, 3 etc

Once cloned, you hit your build script for the new folder (composer install, cache warmup etc) and the final step is to symlink your docroot to the latest folder inside "releases"

If anything goes sideways, you can "rollback" by just symlinking docroot to the previous release folder.

I don't know if you can tell, but I am really excited about all this :)

photo
photo
1

Short back-end updates for everyone:

  • We are updating the version of git available at the command line to git 2.13, the version we ship in /usr/local/cpanel/3rdparty. This version adds support for a few git repo configurations we want to use. If users already have a non-system git in their path, we won't touch it.
  • We're introducing a new namespace in UAPI called VersionControl and designing it in such away that adding support for Mercurial (or, shudders, SVN) should be pretty straightforward to do when/if there's demand for it.

I'm hoping to have a public 'not even EDGE yet' build available for everyone to play with by the end of next week. (Don't hold me to that, lots can happen between now and then which might delay it). Such a build will have the new feature available in cPanel and provide the means to create and list git repositories as well as discover repositories that were created outside of the feature.

photo
1

Update on the release of a 'not even edge yet' build: not happening this week due to having to delay for the targeted security release we issued earlier this week, as well as putting some finishing touches on the creating and listing repositories portions of the feature. We want to include showing the "clone-able" URLs after you create a repository in the first preview release of it so that you can see the whole "create => clone => push/pull" lifecycle.

I'll set the new expectation to middle of next week and I'll have more details about what's in that preview release later on.

photo
6

Alright, here we go. The first incremental delivery of the feature is about to go into our upstream and be built for release in a "not even edge yet" build. Here's a brief about what it can currently do and some known issues to keep in mind:

  • Adds a "Git Version Control and Deployment" feature to the cPanel end-user interface, with access controlled via a checkbox in the Feature Manager
  • Provides the means to create new bare git repositories on your cpanel end-user account, with the means of specifying the file location in one's home directory where the git repo should go
  • Provides the SSH clone URL for each repository you create so that you can clone those repositories
  • Enables the full "clone, push, pull" development lifecycle once your SSH Access and SSH Keys are configured correctly
  • Provides the means to scan your home directory for git repositories that might already exist that the feature doesn't already know about
  • Shows a searchable sortable paginate-able list of repositories
  • Establishes a new VersionControl namespace of APIs for UAPI. The create method is already capable of creating a git repository by cloning from a third-party though the UI for doing so isn't ready yet.

Feature index page

6dea54ded08e5e08cbbcb0dd04804250

Repository Creation screen

c79a40258f4d98d123481b1b3741f66d

Some known issues:

  • Access to the feature currently requires the end-user to have SSH access enabled for their account and an optional valid authorized SSH key in place. We have plans to remove this restriction in future iterations.
  • If you delete a repository manually and hit 'scan', your deleted repository will remain in the list.

  • Clone URLs are composed using your primary domain, not any addon or subdomains. We aren't sure if this is a bad thing or not, but FYI.
  • Clicking on the 'Deployment Path' will launch the file manager at that location. Right now, because we're supporting only bare repositories, this means end-users will be dropped into a .git folder itself. If they muck about in here, it can potentially damage git"s version history enough to cause repository corruption. The deployment path link will eventually lead them directly into their deployed working tree once the deployment scope gets started.
  • The feature icon is a placeholder. Expect a better one soon!

I"ll have build information available for everyone once the merge and build process finishes up.

Next up, we dive head-long into the deployment related scope of the project!

photo
2

Until it's possible for users to choose the folder where their main domain points to, we create "fake" main domains that don't point anywhere, and create their main domain as an addon domain. Enforcing the use of the main domain in the final version would make that part of the tool unusable.

The use of a fake main domain applies mostly to users who are likely to use git on a daily basis. I don't think we have many users who use git and work in the public_html folder.

photo
1

Could git be added to the limited shell for this to work? Another option with this option is setting the key only work for that set of commands.

photo
1

@denver - that is currently the plan, however we're operating under the assumption that the user has shell access for the first round of implementation. We hope to make this happen before we call it done!

photo
photo
4

For those interested in installing a "not even edge yet" build and playing with this feature, please hit me up! adam.french@cpanel.net - I've got to send you install instructions!

photo
2

We were actually able to get the "clone from external repo" stable for this "not even edge yet" build as well. Here's an updated screenshot of the create page now:

b6dd09361f7dfd9e6b46e9376e112db2

Known Issues:

  • We designed it to work with http/s URLs but it *should* work with ssh:// clone urls as well. If you want to do this, make sure to generate an SSH public key for your server and offer it to wherever the third-party repo is located as authentication.
  • By default, we "check out" the master branch. In future iterations we will provide the means to specify which branch you would like to deploy.

photo
1

Small update for everyone - our release team is updating git to 2.14.0 in the "not even edge yet" builds. If you're curious what's new between 2.13 and 2.14: https://github.com/git/git/blob/master/Documentation/RelNotes/2.14.0.txt - Doesn't really have much of an effect for our particular project, but figured I'd FYI anyway.