I Failed the Turing Test

Redirect Console Output: stdout and stderr

Redirect Console Output: stdout and stderr

Updated on Published on Author VintaCategory: DevOps

File descriptors:

  • 0: stdin
  • 1: stdout
  • 2: stderr
# redirect stdout to a file test.log
$ strace uptime > test.log
$ strace uptime 1> test.log

# redirect stderr to a file test.log
$ strace uptime 2> test.log

# redirect stderr to stdout
# &1 references the value of the file descriptor 1 (stdout)
$ sh /home/vinta/do_shit.sh > /tmp/do_shit.log 2>&1

# 2> 1 actually means that you redirect stderr to a file named 1
$ sh /home/vinta/do_shit.sh > /tmp/do_shit.log 2> 1
$ cat 1

# redirect both stderr and stdout to file test.log
$ strace uptime &> test.log

# run a command in background
$ long-running-command &

ref:
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html
https://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
https://www.brianstorti.com/understanding-shell-script-idiom-redirect/

Lazy evaluation in Django middlewares

Lazy evaluation in Django middlewares

Updated on Published on Author VintaCategory: Python, Web Development

Attach a lazily evaluated function as a property of request in a middleware.

from django.contrib.gis.geoip import GeoIP
from django.utils.functional import SimpleLazyObject

def get_country_code(request):
    g = GeoIP()
    location = g.country(request.META['REMOTE_ADDR'])
    country_code = location.get('country_code', 'TW')

    return country_code

class CountryAndSiteMiddleware(object):

    def process_request(self, request):
        request.COUNTRY_CODE = SimpleLazyObject(lambda: get_country_code(request))

Then you could use request.COUNTRY_CODE whenever you want.

Setup Jupyter and other Machine Learning tools on macOS

Setup Jupyter and other Machine Learning tools on macOS

Updated on Published on Author VintaCategory: AI, Python

Jupyter Notebook is an interactive environment for running code in the browser. It allows you to create interactive documents that contain live code, rich text elements and visualizations. It's also a widely used tool for Data Scientists to make prototypes or demonstrations.

ref:
http://jupyter.org/

Install

$ brew install freetype gcc libffi libpng openssl pkg-config
$ pip install -U 
  cython 
  numpy 
  scipy 
  matplotlib 
  bokeh 
  seaborn 
  scikit-learn 
  surprise 
  gensim 
  nltk 
  pandas 
  jupyter

$ pip install jupyter_contrib_nbextensions && 
  jupyter contrib nbextension install --user

# install kernels for Python 2 and 3
$ python2 -m pip install ipykernel && 
  python2 -m ipykernel install --user

# list kernels
$ jupyter kernelspec list

# remove kernel
$ jupyter kernelspec uninstall apache_toree_scala

# start your notebook server
$ jupyter notebook
$ jupyter notebook --ip 0.0.0.0 --allow-root --no-browser

ref:
https://ipython.readthedocs.io/en/latest/install/kernel_install.html
https://jupyter.readthedocs.io/en/latest/running.html#running
https://github.com/ipython-contrib/jupyter_contrib_nbextensions

Or you could just download Anaconda and install it.
https://www.continuum.io/downloads#osx

Configuration

# ~/.ipython/profile_default/ipython_config.py    
c = get_config()

c.InteractiveShell.ast_node_interactivity = 'all'

# c.InteractiveShellApp.matplotlib = 'notebook'
c.InteractiveShellApp.matplotlib = 'inline'

Usage

Automatic module reload

%load_ext autoreload
%autoreload 2
import your_module

ref:
https://blog.3blades.io/jupyter-notebook-little-known-tricks-b0866a558017

Show media in notebook

# show image
from IPython.display import Image
Image('iris.png')

# show pdf
from IPython.display import IFrame
IFrame('iris.pdf', width='100%', height=700)

Django integration with django-extension

$ python manage.py shell_plus --notebook

ref:
https://stackoverflow.com/questions/35483328/how-do-i-set-up-jupyter-ipython-notebook-for-django

IPython: the Python REPL interpreter

IPython: the Python REPL interpreter

Updated on Published on Author VintaCategory: Python

IPython is a neat alternative to Python's built-in REPL (Read-Eval-Print Loop) interpreter, also a kernel of Jupyter.

ref:
https://ipython.org/
https://jupyter.org/

Useful Commands

# cheatsheet
%quickref

# show details of any objects (including modules, classes, functions and variables)
random?
os.path.join?
some_variable?

# show source code of any objects
os.path.join??

# show nothing for a function that is not implemented in Python
len??

# run some shell commands directly in IPython
pwd
ll
cd
cp
rm
mv
mkdir new_folder

# run any shell command with ! prefix
!ls
!ping www.google.com
!youtube-dl

# assign command output to a variable
contents = !ls
print(contents)

ref:
http://ipython.readthedocs.io/en/stable/interactive/tutorial.html

Magic Functions

# list all magic functions
%lsmagic

# run a Python script and load objects into current session
%run my_script.py

# run a profiling for multi-line code
%%timeit
array = []
for i in xrange(100):
    array.append(i)

# paste multi-line code
# you might not need them in IPython 5.0+, just paste your code
%cpaste

# explore objects
%pdoc some_object
%pdef some_object
%psource some_object
%pfile some_object

# if you call it after hitting an exception, it will automatically open ipdb at the point of the exception
%debug

# the In object is a list which keeps track of the commands in order
print(In)

# the Out object is a dictionary mapping input numbers to their outputs
print(Out)
print(Out[2], _2)

ref:
https://ipython.readthedocs.io/en/stable/interactive/magics.html
https://www.safaribooksonline.com/library/view/python-data-science/9781491912126/ch02.html

Issues

Reload Any Python Module

from your.project import your_module
your_module.run_shit(123)

# after you made some changes on your_module
from importlib import reload
reload(your_module)

your_module.run_shit(123)

ref:
https://stackoverflow.com/questions/5364050/reloading-submodules-in-ipython

Speed up Python and Node.js builds on Travis CI

Speed up Python and Node.js builds on Travis CI

Updated on Published on Author VintaCategory: DevOps, JavaScript, Python

Travis CI's caching archives all directories listed in the configuration and uploads them to Amazon S3. Cached contents are available to any build on the repository, including Pull Requests. For Python and Node.js projects, you could cache both site-packages and node_modules directories in every Travis CI build.

Here is an example of .travis.yml:

sudo: false

language: python

python:
  - "2.7"

node_js: 4

cache:
  directories:
    - $HOME/.cache/pip
    - $HOME/virtualenv/python2.7.9/lib/python2.7/site-packages
    - node_modules

before_install:
  - pip install -U pip

install:
  - pip install -r requirements.txt
  - pip install coverage --ignore-installed
  - npm install

script:
  - coverage run manage.py test

In my case, after applying these changes, the installation time of pip and npm reduces from 180 seconds to 5 seconds.

One thing should be mentioned here: Since we didn't specify any bin folder in the configuration (and I don't think that's necessary), any executable file that is installed by pip such as coverage or django-admin.py will not exist in subsequent builds. If you need those commands, you could just force install them by adding pip install some_package --ignore-installed.

ref:
https://docs.travis-ci.com/user/caching/
https://stackoverflow.com/questions/19422229/how-to-cache-requirements-for-a-django-project-on-travis-ci
https://tzangms.com/how-to-speed-up-python-unit-test-on-travis-ci/