Skip to main content

Python

warning

The Python runtime is in BETA.

The Python runtime allows you to run Python code in AutoKitteh.

note

The minimal Python version we support is 3.11. In order to run Python workflow, Python must be installed on the machine running the code - where the ak command is installed. ak will use the first Python it finds in the PATH.

Example​

So you can jump right ahead.

Write the following two files in a directory.

autokitteh.yaml
---
version: v1

project:
  name: py_simple
  connections:
    - name: http_trigger
      integration: http
  triggers:
    - name: post
      connection: http_trigger
      event_type: post
      call: simple.py:greet
  vars:
    - name: USER
      value: Garfield
simple.py
from os import getenv
import json

HOME = getenv('HOME')  # From environment
USER = getenv('USER')  # From "vars" section in the manifest


def greet(event):
    print(f'INFO: simple: HOME: {HOME}')
    print(f'INFO: simple: USER: {USER}')

    print(f'INFO: simple: event: {event!r}')
    body = event['data']['body']
    print(f'BODY: {body!r}')
    request = json.loads(body)
    print(f'REQUEST: {request!r}')

Start AutoKitteh

ak up --mode dev

Deploy (in the directory where you have the above files

ak deploy -m ./autokitteh.yaml -d .

Now you can run the workflow:

curl -i -X POST -d '{"user": "joe", "event": "login"}' http://localhost:9980/http/py_simple/

Any output from Python to standard output (sys.stdout) or standard error (sys.stderr) is captured in the session logs. Which means both print and calls to the logging module.

Use the session log command to view:

ak session log --prints-only

Overview​

The Python runtime can receive events from configured connection triggers. It cannot use the connection for outgoing messages, you will need to use a Python library to do that.

Handler Functions​

Each Python handler is a function that receives a single parameter called event. The event has several fields:

  • data: This is the "raw" event sent by the trigger. The content of data depends on the trigger's integration.

The event is a dict, but you can also use attribute access (.) with it. Both of the following lines are valid:

body = event['data']['body']
body = event.data.body
note

You cannot set event values with attributes. The following is valid:

event['data']['user'] = 'Garfield'

The following will raise an exception:

event.data.user = 'Garfield'

:::

Module Level Function Calls​

Functions that are called during module import are executed as regular functions (and not activities).

For example:

from os import getenv

api_key = getenv('API_KEY')

getenv will not run as activity. This is intentional and useful for setting up connections and other global state.

Third Party Dependencies​

AutoKitteh creates a virtual environment for the system Python. This will be the first python3 or python found in your PATH environment variable.

note

The virtual environment is created once on the first time you deploy a Python workflow. This will cause the first time you deploy a Python workflow to take a while. Next deployments will be faster.

Once the virtual environment is created, AutoKitteh will use the python executable inside it.

To install packages at the virtual environment, see here.

info

You can override which Python executable to use by setting the AK_WORKER_PYTHON environment variable. If you set this environment variable, AutoKitteh will this this Python without creating a virtual environment.

If you want to create your own virtual environment, see here. To see how you can install other version of Python, see pyenv.

The virtual environment is preloaded with the following packages:

requirements.txt
# Integrations

atlassian-python-api ~= 3.41
boto3 ~= 1.34
google-api-python-client ~= 2.122
google-auth-httplib2 ~= 0.2
google-auth-oauthlib ~= 1.2
jira ~= 3.8
openai ~= 1.14
PyGithub ~= 2.2
redis ~= 5.0
slack-sdk ~= 3.27
twilio ~= 9.0

# AutoKitteh SDK

autokitteh ~= 0.2

# General

beautifulsoup4 ~= 4.12
PyYAML ~= 6.0
requests ~= 2.31

See the update list here.

Installing Python Packages​

If you want to install other packages, first find out where the virtual environment is.

ak config where

Say the "Data home directory" is /home/ak/.local/share/autokitteh, then the virtual environment is at /home/ak/.local/share/autokitteh/venv.

To install a package (e.g. pandas), run Python from the virtual environment:

/home/ak/.local/share/autokitteh/venv/bin/python -m pip install pandas~=2.2

Local Development​

Setting Up Your IDE​

If you want to debug your code and get autocompletion, you need to tell your IDE to use the Python from the autokitteh virtual environment. This way, your code will run in the same environment that AutoKitteh uses to run it.

If you don't have AutoKitteh virtual environment, you can create a virtual environment yourself.

Download requirements.txt to your machine. The in a terminal, run:

python -m virtualenv venv
venv/bin/python -m pip install -r requirements.txt

This will create a virtual environment called venv, point your IDE Python to /path/to/venv/bin/python.

Visual Studio Code​

Copy the interpreter path (see above) to the clipboard.

Open the command pallet and write Python: Select Interpreter and paste the interpreter path. For more information, see the official Visual Studio Code documentation.

Also, don't forget to install the autokitteh extension so you'll be able to install and manage workflows.

PyCharm​

In Settings pick your project and then Python Interpreter. Click on Add interpreter on the top right and select Add Local Interpreter. Click on System Interpreter from the list on the left and then enter the interpreter path from autokitteh virtual environment (see above).

For more information, see the official PyCharm documentation.

Isolated Handler Functions​

Handler functions are the functions you define in the manifest, they are called by AutoKitteh on new events. Handler function are regular Python functions, and you can debug them like you debug other Python code.

If your handler function is using vars or secrets, you need to set the environment before importing the handler module. See Working with Secrets on how to name your environment variables.

For example, say you have a handler.py with an HTTP on_event handler function:

handler.py
def on_event(event):
    print("BODY:", event.data.body.decode())

Then you can run the handler function like this:

from autokitteh import AttrDict
import handler

event = AttrDict({"data": {"body": "hello".encode()}})
handler.on_event(event)
note

AutoKitteh events are an instances of AttrDict, this is why we create event above in such manner.

python -m pip install autokitteh.

You can see the API documentation here. The autokitteh module also contains common operation for connection to various services.

Debugging Workflows​

Before you run ak locally, set up any environment variables for secrets/vars that are required by your workflow.

export TOKEN=s3cr3t

Another option is to use the ak var set command instead of setting environment variables. You need to run this command after starting ak

Then run

ak up --mode dev

The easiest way to run a workflow locally is to add an HTTP trigger.

autokitteh.yaml
version: v1

project:
  name: hello
  connections:
    - name: http_trigger
      integration: http
      token: event
  triggers:
    - name: events
      connection: http_trigger
      event_type: post
      entrypoint: handler.py:on_event

Next, deploy your workflow:

ak deploy --manifest ./autokitteh.yaml --file handler.py

And now you can trigger your code:

curl -d hello http://localhost:9980/http/hello/events

You can view the print output using the ak session log command:

ak session log --prints-only

Every time you change the handler code, you need to re-deploy the workflow.

Limitations​

danger

We try to lift these limitations as we progress, but currently your workflow will fail if you don't follow them.

Function Arguments and Return Values Must Be Pickleable​

We use pickle to pass function arguments back to AutoKitteh to run as an activity. See What can be pickled and unpickled? for supported types. Most notably, the following can't be pickled:

  • lambda
  • dynamically generate functions (notably os.environ.get)
bad.py
import db

def handler(event):
    mapper = lambda n: n.lower()
    db.apply(mapper)  # BAD
good.py
import db

def mapper(n):
    return n.lower()

def handler(event):
    db.apply(mapper)  # GOOD

Using the autokitteh.activity Decorator​

The autokitteh.activity decorator allow you to mark a function so that it'll always run as activity. This allows you to run function with arguments or return values that are not compatible with pickle.

note

The autokitteh module is installed to the default AutoKitteh virtual environment. If you provide your own Python by setting the AK_WORKER_PYTHON environment variable, you will need to install the autokitteh module.

python -m pip install autokitteh.

You can see the API documentation here. The autokitteh module also contains common operation for connection to various services.

Say you have the following code in your handler:

import json
from urllib.request import urlopen


def handler(event):
    login = event['login']
    url = f'https://api.github.com/users/{login}'
    with urlopen(url) as fp:
        resp = json.load(fp)
    print('user name:', resp['name'])

If you try to run this handler it will fail since the result of urlopen cannot be pickled. What you can do is move the code into a function marked as activity:

import json
from urllib.request import urlopen

import autokitteh


def handler(event):
    login = event['login']
    info = user_info(login)
    print('user name:', info['name'])


@autokitteh.activity
def user_info(login):
    url = f'https://api.github.com/users/{login}'
    with urlopen(url) as fp:
        resp = json.load(fp)
    return resp

All the code in user_info will run in a single activity. Since user_info accepts a str and returns a dict it can run as activity.

How It Works​

AutoKitteh will first load your code and instrument every function call that is external to your module. Each instrumented call will be converted to an activity.

Working with Secrets​

AutoKitteh has a secret store. These secrets are exposed to Python via the environment.

For example, if you set a token:

$ ak var set --secret --env env_01hyg78h31e3yahn2dt4mf9nzz TOKEN s3cr3t
tip

To get the list of environments (for --env), use ak env list.

Then in your python code you can write the following to access it:

from os import getenv

def handler(event):
    token = getenv('TOKEN')
    if not token:
        raise ValueError('missing `TOKEN` environment variable')
    # Use token...

If you need a secret in a file (e.g. Google's service account keys), you can set the secret value to be the file's contents. Download the file (e.g. to /tmp/credentials.json), and run these commands:

$ ak var set --secret --env env_01hyg78h31e3yahn2dt4mf9nzz GOOGLE_CREDS_DATA < /tmp/credentials.json
$ shred -u /tmp/credentials.json  # Delete local file, optional but recommended

Then in your script you can write the file and set GOOGLE_APPLICATION_CREDENTIALS environment variable:

import os


def ensure_google_credentials():
    env_key = 'GOOGLE_APPLICATION_CREDENTIALS'
    if os.getenv(env_key):
        return

    creds_file = 'credentials.json'
    data = os.getenv(env_key)
    with open(creds_file, 'w') as out:
        out.write(data)
    os.putenv(env_key, creds_file)


def commit_handler(event):
    ensure_google_credentials()
    ...  # Rest of handler code

Connection Secrets​

Connections have their own secrets. You can access them via the environment as well. The environment variable name is the connection name followed by two underscores and the secret name.

For example: Say you defined an HTTP connection called http_trigger and set its bearer token to s3cr3t.

You can look at the connection web page (under /connections) to see the environment variable name under the Vars section. In this case, it'll be auth. So in your workflow code you can write:


token = getenv('http_trigger__auth')
print(f'HTTP token: {token!r}')

Note that the value of token is Bearer s3cr3t, not s3cr3t.