Python: Protip - Enable Tab Completion in Python Shell

I was recently chatting with someone that made the statement; “The only reason why I use my text editor is because of tab completion, I am a minimalist pythonista.”. I responded with, “If your a minimalist, why don’t you have tab completion enabled in the interpreter?” They were unaware of the ability to enable tab completion within python. Here is a my PSA on how to enable this behavior. :)

By following these instructions you will be able to enable tab-completion within your python shell.

Preperation steps:

Before enabling tab-completion, you may need to install 2 python modules (rlcompleter, readline). While these libraries are mostly included with python2.6+, some versions (OS X, for example), require the updated version to allow readline to function correctly.

```bash shell

pip install readline pip install rlcompleter

```

Step 1:

If you don’t already have a ~/.pyrc file, this command will create one for you, which is required for this trick to work.

```bash shell

> touch ~/.pyrc

```

Step 2:

Now we will create a file within your homedir which will instruct python to bind tab completion at python launch.

```python ~/.pyrc

import rlcompleter import readline readline.parse_and_bind(“tab: complete”)

```

Step 3:

Now to ensure that your newly created ~/.pyrc file is executed each time python starts, add the following to your ~/.bashrc (or equiv. shell rc).

```bash shell

> export PYTHONSTARTUP=“[PATH TO PYRC FILE]/.pyrc”

*OR TO MAKE THE CHANGE PERSIST TERMINAL CLOSURE

> echo export PYTHONSTARTUP=“[PATH TO PYRC FILE].pyrc” >> ~/.bashrc

```

Now to test, execute the following:

```bash shell

> source ~/.bashrc #reloads your ~/.bashrc file (if you added the entry to your ~/.bashrc, else ignore)

```

Now we will open the actual python shell, and view the tab completion goodness.

```python shell

Python 2.7.6 (v2.7.6:3a1db0d2747e, Nov 10 2013, 00:42:54) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type “help”, “copyright”, “credits” or “license” for more information.

import os os. Display all 234 possibilities? (y or n) os.EX_CANTCREAT os.class( os.fchdir( os.putenv( os.EX_CONFIG os.delattr( os.fchmod( os.read( os.EX_DATAERR os.dict os.fchown( os.readlink( os.EX_IOERR os.doc os.fdopen( os.remove( os.EX_NOHOST os.file os.fork( os.removedirs( os.EX_NOINPUT os.format( os.forkpty( os.rename( os.EX_NOPERM os.getattribute( os.fpathconf( os.renames( os.EX_NOUSER os.hash( os.fstat( os.rmdir( os.EX_OK os.init( os.fstatvfs( os.sep os.EX_OSERR os.name os.fsync( os.setegid( os.EX_OSFILE os.new( os.ftruncate( os.seteuid( os.EX_PROTOCOL os.package os.getcwd( os.setgid( os.EX_SOFTWARE os.reduce( os.getcwdu( os.setgroups( os.EX_TEMPFAIL os.reduce_ex( os.getegid( os.setpgid( os.EX_UNAVAILABLE os.repr( os.getenv( os.setpgrp( os.EX_USAGE os.setattr( os.geteuid( os.setregid( os.F_OK os.sizeof( os.getgid( os.setreuid( os.NGROUPS_MAX os.str( os.getgroups( os.setsid( os.O_APPEND os.subclasshook( os.getloadavg( os.setuid( os.O_ASYNC os.copy_reg os.getlogin( os.spawnl( os.O_CREAT os.execvpe( os.getpgid( os.spawnle( os.O_DIRECTORY os.exists( os.getpgrp( os.spawnlp( os.O_DSYNC os.exit( os.getpid( os.spawnlpe( os.O_EXCL os.get_exports_list( os.getppid( os.spawnv( os.O_EXLOCK os.make_stat_result( os.getsid( os.spawnve( os.O_NDELAY os.make_statvfs_result( os.getuid( os.spawnvp( os.O_NOCTTY os.pickle_stat_result( os.initgroups( os.spawnvpe( os.O_NOFOLLOW os.pickle_statvfs_result( os.isatty( os.stat( os.O_NONBLOCK os.spawnvef( os.kill( os.stat_float_times( os.O_RDONLY os.abort( os.killpg( os.stat_result( os.O_RDWR os.access( os.lchflags( os.statvfs( os.O_SHLOCK os.altsep os.lchmod( os.statvfs_result( os.O_SYNC os.chdir( os.lchown( os.strerror( os.O_TRUNC os.chflags( os.linesep os.symlink( os.O_WRONLY os.chmod( os.link( os.sys os.P_NOWAIT os.chown( os.listdir( os.sysconf( os.P_NOWAITO os.chroot( os.lseek( os.sysconf_names os.P_WAIT os.close( os.lstat( os.system( os.R_OK os.closerange( os.major( os.tcgetpgrp( os.SEEK_CUR os.confstr( os.makedev( os.tcsetpgrp( os.SEEK_END os.confstr_names os.makedirs( os.tempnam( os.SEEK_SET os.ctermid( os.minor( os.times( os.TMP_MAX os.curdir os.mkdir( os.tmpfile( os.UserDict os.defpath os.mkfifo( os.tmpnam( os.WCONTINUED os.devnull os.mknod( os.ttyname( os.WCOREDUMP( os.dup( os.name os.umask( os.WEXITSTATUS( os.dup2( os.nice( os.uname( os.WIFCONTINUED( os.environ os.open( os.unlink( os.WIFEXITED( os.errno os.openpty( os.unsetenv( os.WIFSIGNALED( os.error( os.pardir os.urandom( os.WIFSTOPPED( os.execl( os.path os.utime( os.WNOHANG os.execle( os.pathconf( os.wait( os.WSTOPSIG( os.execlp( os.pathconf_names os.wait3( os.WTERMSIG( os.execlpe( os.pathsep os.wait4( os.WUNTRACED os.execv( os.pipe( os.waitpid( os.W_OK os.execve( os.popen( os.walk( os.X_OK os.execvp( os.popen2( os.write( os._Environ os.execvpe( os.popen3(
os.all os.extsep os.popen4(
os.

```

In the above example, I have imported the “os” module, typed “os.” and have pressed . Now all of the possible matched object names available within the module are shown. viola, python tab completion enabled.

Python: Recursion, Part Duex

I ended up on a roll tonight writing my documentation on python recursive functions and decided to continue writing the second portion of this blog entry ,first post found here. In this post I plan to provide you with a slightly more complicated example of a recursive function while also showing a side-by-side comparison of recursion vs. iteration (Spoiler alert, iteration wins!).

Italians and integers, breeding like rabbits!

My example algorithm that I am introducing for advanced recursive method usage is none other than the Fibonacci number.

The Fibonacci numbers are a sequence of the following integer values:

[0,1,1,2,3,5,8,13,21,34,55,89,144 …]

The Fibonacci numbers are defined by the equation: ```text equation

Fn = Fn-1 + Fn-2 with F0 = 0 and F1 = 1

```

The Fibonacci sequence(numbers) are named after the mathematician Leonardo of Pisa, who is better known as Fibonacci. In his book “Liber Abaci” (published 1202) he introduced the sequence as an exercise dealing with (biologically unrealistic) rabbit breeding habits. His sequence of the Fibonacci numbers begins with F1 = 1, while in modern mathematics the sequence starts with F0 = 0. But this has no effect on the other members of the sequence.

OK, now that you are familiar with the introduction of Fionacci numbers lets get to the gooey, nerdy center of this example! ☺

Solving for Fibonacci sequencing in Python

The Fibonacci numbers are the result of an artificial rabbit population, satisfying the following conditions:

A newly born pair of rabbits, one male, one female, build the initial population. The rabbits are able to successfully mate at the age of one month. This means that at the end of the second month of life, the female rabbit gives birth to 2 “hoppy”(harhar), healthy rabbits. The new sibling pair of rabbits consist of 1 male and one female. Did I mention that all of the rabbit spawn are immortal?!? Every rabbit from here on out will never die and just keep producing offspring. What a job, eh? Every new pair of male|female rabbits will continue to mate after the second month of life until infinity. The Fibonacci numbers are the numbers of rabbit pairs after n months, i.e. after 10 months we will have F10 rabbits.

The Fibonacci equation is very easy to program, as your equation depicted within the code is almost 1:1 with the original equation:

```python Fibonacci recursive example

def fibonacci_recursive(n):

if n == 0:
    return 0
elif n == 1:
    return 1
else:
    return fibonacci_recursive(n-1) + fibonacci_recursive(n-2)

```

The above example depicts the Fibonacci numbers solution by using recursive methods. Now I will provide an example of a python function that returns the same Fibonacci numbers only using iteration instead of recursion.

```python Finbonacci iterative example

def fibonacci_iterative(n):

a, b = 0, 1
for i in range(n):
    a, b = b, a + b
return a

```

If you try both of these functions in your python interpreter you will notice that fibonacci_iterative method is “Orders of magnitude” faster than the fibonacci_recursive equivalent.

Why is recursion so slow?!

In Java, C, and Python, recursion is fairly expensive compared to iteration (in general) because it requires the allocation of a new stack frame. In some C compilers, one can use a compiler flag to eliminate this overhead, which transforms certain types of recursion (actually, certain types of tail calls) into jumps instead of function calls.

In functional programming language implementations, sometimes, iteration can be very expensive and recursion can be very cheap. In many, recursion is transformed into a simple jump, but changing the loop variable (which is mutable) sometimes requires some relatively heavy operations, especially on implementations, which support multiple threads of execution. Mutation is expensive in some of these environments because of the interaction between the mutator and the garbage collector, if both might be running at the same time.

There are practical ways that we can help along recursive functions in order to speed them up however. Lets move onto the race between recursion vs. iteration and then we can describe how to speed up our recursion functions in this example.

The great race

In the example below we are going to place both of our above Fibonacci functions into an importable python file. Then we are going to write a new script that imports / executes both functions while performing timing calculations comparing the execution times.

```python import_me.py

def fibonacci_iterative(n):

a, b = 0, 1
for i in range(n):
    a, b = b, a + b
return a

def fibonacci_recursive(n):

if n == 0:
    return 0
elif n == 1:
    return 1
else:
    return fibonacci_recursive(n-1) + fibonacci_recursive(n-2)

```

Now we will write a python script that will allow for us to import and execute the two functions within “import_me.py” and measure the execution times.

```python lets_race.py

from timeit import Timer from import_me import fibonacci_recursive

t1 = Timer(“fibonacci_recursive(10)”,“from import_me import fibonacci_recursive”)

for i in range(1,41):

#Import, execute and time recursive function.
s = ("fibonacci_recursive(" + str(i) + ")")
t1 = Timer(s,"from import_me import fibonacci_recursive")
time1 = t1.timeit(3)

#Import, execute and time iterative function. 
s = ("fibonacci_iterative(" + str(i) + ")")
t2 = Timer(s,"from import_me import fibonacci_iterative")
time2 = t2.timeit(3)

print("n=%2d, recursive exec time: %8.6f, iterative exec time:  %7.6f, iterative percent faster: %10.2f" % (i, time1, time2, time1/time2))

```

text output n= 1, recursive exec time: 0.000005, iterative exec time: 0.000007, iterative percent faster: 0.72 n= 2, recursive exec time: 0.000007, iterative exec time: 0.000007, iterative percent faster: 1.00 n= 3, recursive exec time: 0.000008, iterative exec time: 0.000007, iterative percent faster: 1.14 n= 4, recursive exec time: 0.000011, iterative exec time: 0.000008, iterative percent faster: 1.35 n= 5, recursive exec time: 0.000017, iterative exec time: 0.000008, iterative percent faster: 2.09 n= 6, recursive exec time: 0.000026, iterative exec time: 0.000008, iterative percent faster: 3.21 n= 7, recursive exec time: 0.006997, iterative exec time: 0.000016, iterative percent faster: 438.01 n= 8, recursive exec time: 0.000060, iterative exec time: 0.000009, iterative percent faster: 6.63 n= 9, recursive exec time: 0.000093, iterative exec time: 0.000010, iterative percent faster: 9.29 n=10, recursive exec time: 0.000148, iterative exec time: 0.000010, iterative percent faster: 14.79 n=11, recursive exec time: 0.000236, iterative exec time: 0.000010, iterative percent faster: 23.57 n=12, recursive exec time: 0.000381, iterative exec time: 0.000010, iterative percent faster: 38.05 n=13, recursive exec time: 0.007792, iterative exec time: 0.000012, iterative percent faster: 653.64 n=14, recursive exec time: 0.001105, iterative exec time: 0.000012, iterative percent faster: 92.70 n=15, recursive exec time: 0.008806, iterative exec time: 0.000012, iterative percent faster: 724.22 n=16, recursive exec time: 0.010133, iterative exec time: 0.000017, iterative percent faster: 598.61 n=17, recursive exec time: 0.019284, iterative exec time: 0.000018, iterative percent faster: 1064.25 n=18, recursive exec time: 0.022575, iterative exec time: 0.000017, iterative percent faster: 1333.61 n=19, recursive exec time: 0.056988, iterative exec time: 0.000020, iterative percent faster: 2845.54 n=20, recursive exec time: 0.079930, iterative exec time: 0.000019, iterative percent faster: 4243.67 n=21, recursive exec time: 0.114535, iterative exec time: 0.000018, iterative percent faster: 6405.25 n=22, recursive exec time: 0.157802, iterative exec time: 0.000018, iterative percent faster: 8708.80 n=23, recursive exec time: 0.261647, iterative exec time: 0.000018, iterative percent faster: 14439.83 n=24, recursive exec time: 0.425641, iterative exec time: 0.000018, iterative percent faster: 23803.57 n=25, recursive exec time: 0.685106, iterative exec time: 0.000019, iterative percent faster: 35919.29 n=26, recursive exec time: 1.090093, iterative exec time: 0.000018, iterative percent faster: 60962.41 n=27, recursive exec time: 1.777188, iterative exec time: 0.000019, iterative percent faster: 93175.84 n=28, recursive exec time: 2.874248, iterative exec time: 0.000020, iterative percent faster: 143517.50

In part three of this blog series I will cover in greater detail why the recursive solution is slower than the iterative solution. I will also cover some tips / tricks that we can do to help tune the recursive function in order to make it perform much like its iterative counterpart.

-g

Messaging Queue Showdown: Amazon SQS vs Celery(RabbitMQ)

As I am a previous engineer at Amazon AWS I have plenty of experience with AWS services and have exclusively leveraged SQS in many projects. I have always had a mostly positive experience with Amazon AWS SQS https://aws.amazon.com/sqs/ however I have been curious for quite some time what the benefit of rabbitMQ http://www.rabbitmq.com/ could be vs. SQS. Message queuing is not a new concept and has existed for years within computer science/engineering see: http://en.wikipedia.org/wiki/Message_queue if you are unfamiliar with the concept.

*I would like to firstly point out that my experiences with both services have been with Python, however I did create a test instance and performed the same tests with Java yielding the same results.

So I have to start out by saying that Amazons offering for queuing services is fantastic, relatively cheap and easy. The community has also built some fantastic programmatic libraries to interact with the web services. So now you may ask, why look at other services? Well there can be some possible drawbacks of SQS as well that can cause you some problems depending on the situation.

One drawback of SQS implementation is the need for polling of a message queue to determine if new messages have appeared. This is a bit of an issue being as you must now model your application to perform a polling cycle in order to determine if new messages are available, and if so build the logic around consuming the message and popping it out of the queue. You must also be mindful of queue settings that control such things as TTL, maximum message length and which endpoints (if multiple are used) the message is destined for.

Beyond the small issue of having to be responsible for your own polling cycle, you are also billed based upon the number of requests to a queue. I believe the breakdown on billing is something along the lines of 1 million requests == $100.00. While this isn’t a ton of money it can still get quite costly if your distributed app has 100s of queue and must all be polled. Especially if your messages come in bursts, in which case you have a lot of empty queue polling which penalizes you. There are different approaches you can take with SQS to apply back-off algorithms to queuing logic, but the penalty is delay in all other inter-dependent services cascades.

So now enter RabbitMQ.

RabbitMQ is a message queue system based on Erlang and conforming to AMQP (a standard and heavily used message queue protocol). There are obvious overhead in the fact that you must host your own instances of RabbitMQ along with the infrastructure. Also obvious reliability in multi-AZ redundancy will need to be considered unless you continue to host your instance within EC2 and configure appropriately.

So now lets talk about the speed of RabbitMQ, I have one word to describe it. FAST! My testing was performed within Amazon EC2 / AWS within the same region|AZ to be fair. My test was simple, build a queue, spray 5,000 message to the queue as fast as possible, de-queue and discard message. There are many different configurations that RabbitMQ can support such as one-to-one, one-to-many, many-to-many, RPC. In my example I simply used the default which is one-to-one.

So now the “quarter mile” times between the two.

Amazon AWS SQS:
Region: IAD
Service: SQS
Total time: 5:42 minutes:seconds

RabbitMQ EC2:
Region: IAD
Service: EC2
Total time: 0:06 minutes:seconds

What does this prove!?!

Nothing really besides shoving a lot of messages down the pipe and pulling them out on the other side IS indeed faster using RabbitMQ. This does not prove however that the actual service is “better” than SQS by any stretch of the imagination. Both services provide pro/con, and SQS has a rich history of reliable service that affords a great mixture of safe queuing along with a rock solid infrastructure.

On the other hand depending on the configuration of RabbitMQ you can gain a lot of these safeties with seemingly smaller hit in performance in terms of delivery times. I hope to revisit this topic once I have more time to spend, hopefully this helps educate anyone that is posed with approaching these services in the future though.