Category: Technology

If you are a command line junkie, you really need to check out @climagic on twitter.  Some days are better than others but I am constantly amazed at what is possible in bash/csh.  That said, here are a couple commands I have needed recently, many will be worthless to anyone else but oh well:

• • ar vx mypackage.deb – Unpackage a Debian binary install package. The result is actually three tar.gz files
  • dpkg -l  – List all installed Debian packges on a given system.
  • dpkg -c mypacakge.deb  –  List all files provided by the named Debian package.
  • hub pull-request -i 123 -b account/project:master -h account/project:branchtomerge  –  hub is a github utility that allows you to use some github functionality directly from the command line.  The preceding command will issue a pull request for branchtomerge into master and even tie the request to a given issue number (in this case issue #123.)
  • echo $(sha256sum $DEB | cut -f1 -d’ ‘) $(ls -l $DEB | cut -f5 -d’ ‘) $(basename $DEB)  –  This command creates the package hash structured named used INSIDE of Debian changes files.  Using the same command with (sha1sum|md5sum|sha256sum) will provided all three needed package id’s.  The reason this is useful is when you need to recreate a changes file without the original source package.  The rest of the file is fairly straight-forward but the signed package section has to absolutely precise. Also check out this link for more information.
  • asciiquarium  – OK, you might have to install this one first, but it is a full aquarium in ascii characters, including sharks that eat the fish.  Submarines, fishing hooks, and even the lock ness monster.
  • grc tail -f /var/log/maillog –  Note to self, I need to make an rpm for this package.  grc is a generic colorizer for other command line programs that don’t use color by default (like tail, traceroute, syslog, etc…)
  • isohybrid -h 64 -s 32 mycdimage.iso  –  Adds a simple filesystem layout to a standard iso image so it can be written to USB drive as well as an regular CD.  Really useful for building custom Linux CD/USB images.

So, after looking around for an answer today I finally found out where the Debian install CD stores its cd/usb boot menu configuration files.  While I have already had a great deal of experience editing grub.conf files by hand, this methodology simply doesn’t work on an “El Torito” Joliet CDROM image.  So Debian set-up their boot image (as part of the initial ram disk) inside of the /ISO/isolinux/ directory where ISO is the uncompressed version of the boot image.  Specifically you can configure things like:

• • The boot option timeout in /isolinux/isolinux.cfg
  • The background splash image in /isolinux./splash.png (640×480 on the default menu set-up)
  • Which sub-menu’s, options, boot methods, and GUI installs are available via the /isolinux/menu.cfg

Honestly, I may be the only person on the planet trying to figure this stuff out; but here it is for future reference or for anyone who wants to make their very own custom Debian install CD .

I needed to document the process I used to get a private Debian package repository with some custom applications.  Here is the process I went through.  You can find some reference links under my previous articles “Debian” heading.

Building

Basic steps to building setting up your personnel project to build Debian packages.

1. Get most recent software version for your project (git pull, etc..)
2. Rename project folder to include a default project number.  For example $mv myProject myProject-1.0
3. In project directory run dh_make.  This will create a debian diectory with all the necessary files to build a deb package.
4. Now would be a good time to edit your debian/control file and make any changes needed.
5. If you project is a simple copy operation (say a php web application being installed onto an existing apache server) you will want to do the following:
   1. Create and edit a new file under debian/ named myProject.install (replace myProject with the name of your project.)
   2. Edit the file to specify the where you want the files copied to.  The format of the should be something like this (notice you can use wildcards:)
      

      myfolder/bin/* usr/bin
      src/etc/myproject.config etc/
      myfille usr/share/myfolder

   3. Make sure you debian/rules file looks something like this:
      

      #!/usr/bin/make -f

      %:
      dh $@

6. Now, you should be able to build your package (and the changes & dsc files) by running dpkg-buildpackage.  The resulting packages will be in the directory one level up.

Hosting:

Hosting your own git repository can vary in complexity depending on which software you use to actually build the repository.  The easiest one I found to setup was mini-dinstall.  Start by installing mini-dinstall and apache2 via apt.  Apache is configured to start a basic server (all that we need) with the web page files hosted in /var/www.  Just clean out the www directory, add the mini-dinstall folders after the install:

rm -rf /var/www/*
mkdir -p /var/www/mini-dinstall/incoming

You will also need to create a configuration file for mini-dinstall to use when creating the package repository supporting files. Create/edit the file /etc/mini-dinstall.conf with something like the following:

[DEFAULT]
archivedir = /var/www
mail_to =
verify_sigs = false
architectures = amd64
archive_style = simple-subdir
generate_release = true
mail_on_success = false
release_codename = myreponame
release_description = My Repo Name  Hosting
release_label = myrepo
release_origin = myrepo

At this point mini-dinstall could be configured to run in server mode and watch for incoming packages, but using the utility below I have mine configured to run in batch mode every time I put new files on the hosted server.  This will cause mini-dinstall to create a debian package repository structure that can be accessed directly via apt.  Just add something like the following to your /etc/sources.list

deb http://yourserver.com unstable/abd64/

Deploying:

There is a really nice utility called dput that can be used for deploying software packages (and change files) to a hosting server.  The easiest way to get started is to install dput and then setup a configuration file.  Create a file in your home directory called .dput.cf (or globally in /etc/dput.cf) and add a deployment location like this:

[myservername]
fqdn                    = debian.myserverurl.com
method               = scp
incoming            = /var/www/mini-dinstall/incoming
login                    = root
post_upload_command = ssh root@debian.myserverurl.com mini-dinstall -b

That last line creates the repository using mini-dinstall mentioned above in section “hosting”.  This is particularly useful if you already share public keys with the remote system via ssh.  One you have set it up you can do deployment by typing:

dput -u myservername myproject.change

Where myproject.change is the file created above in “Building”.

I am using a RackSpace cloud running Fedora 18 to install GitLab are a replacement for GroundWarp’s Gitolite server.  Here are some install instructions for getting it running on CentOS6/RHEL6 (the commerical systems based on Fedora.) iconv-devel

Dependency Installation:

Start by doing yum install/groupinstall the following packages and any dependencies they find.

• yum install ruby mysql -server git redis ruby-devel
• yum groupinstall “Development Tools”
• yum install mysql libxslt-devel libyaml-devel libxml2-devel gdbm-devel libffi-develzlib-devel openssl-devel libyaml-develreadline-devel curl-devel openssl-devel pcre-develmemcached-devel valgrind-devel mysql-devel ImageMagick-devel ImageMagick libicu libicu-devel libffi-devel rubygem-bundler

Start your database servers and configure them to start on boot.

systemctl start redis.service
systemctl enable redis.service
systemctl start mysqld.service
systemctl enable mysqld.service

MySQL Server Setup:

Start by logging into the mysql shell:

mysql -u root

Then we create the database we will need, and create a user who can edit/manage that database.

create database gitlabdb;
grant usage on *.* to gitlabuser@localhost identified by “inventapasswordhere”;
grant all privileges on gitlabdb.* to gitlabuser@localhost;

Service User and SSH Configuration:

Create the user account that the GitLab service will be running under.  This will also be used to build some of the necessary dependancies. After creating the user switch to that user’s login.

useradd git
passwd git
su -l git

From here forward (unless otherwise specified) make sure you are logged in as the git user.  The remaining configuration is done as that user.

The next steps are required for setting up the SSH keys pairs for your user account (git uses SSH in the background for most of its tasks.)  Choose the defaults, but make sure to supply a paraphrase (not just a password) when prompted by ssh-keygen.

ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

GitLab Shell:

gitlab-shell needs to be installed built/installed for this user.  The following steps sets everything up and builds the packages directly from github (which is the easiest way honestly.)

cd ~
mkdir gitlab-shell
git clone https://github.com/gitlabhq/gitlab-shell.git gitlab-shell/
cd gitlab-shell
cp config.yml.example config.yml

Now you are going to want to use a text editor (nano,vim, etc…) to edit the following change to your config.yml file.

gitlab_url: “http://yourdomainforgitlab.com/”

Now run the gitlab-shell installer

./bin/install

Download & Configure GitLab:

Next we will setup our gitlab folders, download Gitlab, and begin configuring it to our set-up

cd ~
mkdir gitlab-satellites
mkdir gitlab
git clone https://github.com/gitlabhq/gitlabhq.git gitlab.
cd gitlab
git checkout 5-2-stable
mkdir tmp/pids/
mkdir public/uploads/
mkdir tmp/sockets/
chown -R git.git log/ tmp/
chmod -R u+rX log/ tmp/ public/uploads/
cp config/gitlab.yml.example config/gitlab.yml

Next you will need to make changes (again with your text editor) to the newly copied config/gitlab.yml file.

host: git.yourdomain.com
email_from: gitlab-noreply@git.yourdomain.com
support_email: you@yourdomain.com

Database Configuration:

Our next steps involve configuring the GitLab application database interfaces (remember there are two) and creating initial data entries.

cd ~/gitlab
cp config/puma.rb.example config/puma.rb
cp config/database.yml.mysql config/database.yml

Now configure your database file (the database.yml file you just copied) using your text editor and change the following settings under the production header. (Comment out all the lines under development and testing):

database: gitlabdb
username: gitlabuser
password: “YourSuperSecretPasswordFromTheDatabaseSetupAbove”

We need to install a couple Ruby gems and then  initialize the database.  You do this by running the following commands (note the “without” command tells the bundle to NOT install PostgreSQL gems.):

gem install charlock_holmes –version ‘0.6.9.4’
gem install thor –version ‘0.18.1’

gem install rb-inotify
bundle install –deployment –without development test postgres
bundle exec rake gitlab:setup RAILS_ENV=production

Git Environment Setup:

We need to setup a couple of environmental variables for our local git user. Simply type this:

git config –global user.name “GitLab”
git config –global user.email “gitlab-noreply@git.yourdomain.com”

Date string conversion is fairly painless in JavaScript but sometimes the sheer number of options can be a little annoying to remember.  Below is a table of date display/conversion functions generated from

new Date(“2013-02-19T21:03:39.818Z”).

Hopefully this is helpful to someone else who doesn’t want to look up the output of each of these options.  One more note, these are outputs for US locals in the Central time-zone; other locals and other time-zones would very accordingly.

Date Function	Output
toString()	Tue Feb 19 2013 15:03:39 GMT-0600 (CST)
toDateString()	Tue Feb 19 2013
toGMTString()	Tue, 19 Feb 2013 21:03:39 GMT
toISOString()	2013-02-19T21:03:39.818Z
toUTCString()	Tue, 19 Feb 2013 21:03:39 GMT
toTimeString()	15:03:39 GMT-0600 (CST)
toLocaleString()	Tue 19 Feb 2013 03:03:39 PM CST
toLocaleDateString()	02/19/2013
toLocaleTimeString()	03:03:39 PM
toJSON()	2013-02-19T21:03:39.818Z
valueOf()	1361307819818
toSource()	(new Date(1361307819818))

GroundWarp has a new contract to do web based development for an Oklahoma City software firm called Phase2 Interactive.  One of the benefits of the job is a brand new Apple MacBook Air for use on the companies projects.  I have booted OSX just long enough to shrink the main system partition to almost nothing so I could Linux on it.  So far the entire process has been working pretty well and I am really excited about the quality of the Apple hardware.  Below are some of the system specific fixes I needed to make so overcome the non-standard Apple configuration is places.

• Setup and install is fundamentally the same as a normal Linux install (at least for Fedora 18) on a 64 bit OS.  As the MacBook Air doesn’t have a built in CDROM you will need to burn the iso file to a USB stick for the install.  The only thing that might catch you is that some iso-to-usb tools (most notably unetbootin) actually copy the files to the USB drive then create their own custom boot menu.  The new secure EFI boot will not allow this without being signed (which Fedora is) so you will need to create your install USB with good old fashioned dd.  Make sure you use the 64 bit edition.
  

  dd if=../Fedora-18-x86-64-DVD.iso of=/dev/sdb

• Apple obviously hates its users, and feels they should be punished for their insolence.  At least that is the only explanation I can come up with them flipping the default location of the Alt and Meta keys (key locations that pre-date the Apple by a dozen years… it would be like making the keyboard layout standard but swapping the location of the F and G keys.)  Thankfully this is an easy fix if you are using KDE.  Go to you System Settings, choose Input Devices, choose Keyboard, then choose the Advanced tab.  There is an expandable option titled Alt/Win key Behavior, click on the Left Alt is Swapped with Left Win. Select Apply.
• By default Apple maps the F1 through F12 keys to their system function options (things like volume up/down and screen brightness.)  This is probably fine for most people but if you are a keyboard jokey or a programmer you almost certainly us the F keys constantly.  The quick fix for this is to run the following command as root:
  

  echo 2 > /sys/module/hid_apple/parameters/fnmode

• I am not sure what the “right” way to set kernel module parameters on system boot now that Lennart Poettering bastardized evil systemd has infect the entire Linux world (remember when Linux was simple and intuitive) but once I find the official way to set that it I will update this post.
• Got to go, but there is more to come.

Couple other bash tips to help with more robust code.  The main improvement I learned from the previous link is the trap function.  This function lets you cleanup when specific system signals get sent from the OS, like INT (what gets sent to a program when Ctrl+c is typed) and the TERM signal.  A great example is:

trap “rm -f $lockfile; exit” INT TERM EXIT

In this case a lockfile is being removed just before closing a bash script.   You can get a full list of all the different system signals with the kill -l command.

The other major bash tool that I have used without ever really understanding what it did is the eval expression.  If you have ever written a sysinit configuration script, you know that you use eval to basically load/set variables from other subscripts or external files.  The reason eval does this is explained here.  The quick explanation is that eval forces bash to evaluate a second time any code reference passed to it.  So setting bash variables in-line is as easy as:

eval $(LANG=C grep -F “DEVICE=” ifcfg-$i)

One last bash builtin that may come in handy for some, the shopt function sets and displays bash built-in extended capabilities.  See the link for the man page with some examples.

Overall, I am consistently amazed at the power and flexibility of the Linux command line.