Introduction

rumo is a tool that reads tabular data and runs rules on it. It uses the nemo rule engine.

rumo has three parts:

• A Rust library.
• A command-line tool with the name rumo.
• A Python package with the name pyrumo.

What rumo does

rumo can do these tasks:

1. Show the structure of a table (the number of rows, the columns and the data types).
2. Convert a table to RDF in the Turtle format.
3. Run nemo rules on a table or on RDF data.
4. Return the results of the rules as Python data, a pandas DataFrame or a Polars DataFrame.

How rumo runs rules

The diagram shows the data flow:

 pandas / Polars DataFrame ──► Turtle (RDF) ──┐
                                              ├──► nemo rules ──► results
 RDF text or RDF file ────────────────────────┘

rumo converts each DataFrame to RDF triples. The nemo rules then read the triples and calculate new facts. rumo returns these facts as tables.

Terms in this manual

Example

import pandas as pd
from pyrumo import Rumo

df = pd.read_csv("examples/sample.csv")

rules = """
@prefix : <http://example.org/> .
good(?name, ?score) :- data(?r, :name, ?name), data(?r, :score, ?s),
    ?score = DOUBLE(?s), ?score > $GOOD_SCORE .
"""

rumo = Rumo(rules=rules, data=df, predicate="data")
print(rumo.run(params={"GOOD_SCORE": 90}).to_pandas("good"))

The output is:

   name  score
0   Bob   92.0
1  Dave   95.1

Installation

Install the Python package

Use pip to install the package:

pip install pyrumo

The package needs Python 3.9 or a later version. The package also installs Polars.

To use pandas DataFrames, install the pandas extra:

pip install "pyrumo[pandas]"

Install the command-line tool

Use a prebuilt binary

Each GitHub release has prebuilt binaries for these platforms:

To install a binary:

1. Download the archive for your platform from the release page.
2. Extract the archive.
3. Move the rumo file to a directory in your PATH.

Build from the source code

The nemo rule engine needs the nightly Rust toolchain. The file rust-toolchain.toml selects this toolchain automatically.

1. Install Rust with rustup.
2. Clone the repository.
3. In the repository directory, run this command:

cargo install --path . --features rules

Note: If you do not set the rules feature, the rumo rules command is not available.

Cargo features

Command-line tool

The rumo command has three subcommands: describe, convert and rules.

To show the help, run this command:

rumo --help

Show the structure of a CSV file

Use the describe subcommand:

rumo describe --data examples/sample.csv

The output is:

DataFrame: 5 rows × 3 columns
  - name (str)
  - age (i64)
  - score (f64)

Convert a CSV file to Turtle

Use the convert subcommand:

rumo convert --data examples/sample.csv --result-format turtle

Refer to DataFrame to RDF conversion for the output format.

Run a rule file

Use the rules subcommand:

rumo rules --rules examples/sample.rls --data examples/sample.csv --param GOOD_SCORE=90

If you do not set --data, rumo finds the @import files in the directory of the rule file.

If you do not set --output, rumo writes the contents of all @export files to the standard output.

Note: The rules subcommand reads the files that the @import statements name. It does not convert the --data file. To run rules directly on a DataFrame, use the Python Rumo class.

Python: describe and convert

The pyrumo module has three functions for Polars DataFrames.

import polars as pl
import pyrumo

df = pl.read_csv("examples/sample.csv")

Get the structure of a DataFrame

describe returns a text with the number of rows, the columns and the data types:

print(pyrumo.describe(df))

DataFrame: 5 rows × 3 columns
  - name (str)
  - age (i64)
  - score (f64)

print_info writes the same text to the standard output:

pyrumo.print_info(df)

Convert a DataFrame to Turtle

to_turtle returns the DataFrame as a Turtle text:

ttl = pyrumo.to_turtle(df, "http://example.org/", "r")
print(ttl)

prefix : <http://example.org/>

:r0 :name "Alice" ;
    :age 25 ;
    :score 88.5 .
...

The second argument is the prefix IRI. The third argument is the start of the name of each row.

Refer to DataFrame to RDF conversion for the rules of the conversion.

Note: These three functions accept only Polars DataFrames. The Rumo class accepts pandas DataFrames and Polars DataFrames.

Python: run rules with Rumo

The Rumo class runs nemo rules on data. A Rumo object keeps:

• One rule program.
• Zero or more data resources.

Procedure

1. Make a Rumo object.
2. Load the rule program.
3. Load the data.
4. Run the rules.
5. Read the results.

You can do steps 1 to 3 in one call:

from pyrumo import Rumo

rumo = Rumo(rules_file="rules.rls", data=df, predicate="data")

You can also do each step with a method:

rumo = Rumo()
rumo.read_rules_file("rules.rls")
rumo.read_dataframe(df, predicate="data")

Then run the rules and read the results:

results = rumo.run(params={"GOOD_SCORE": 90})
table = results.to_pandas("good")

Use one object more than one time

A Rumo object does not change when you run the rules. You can run the rules again with different parameters:

high = rumo.run(params={"GOOD_SCORE": 90})
low = rumo.run(params={"GOOD_SCORE": 50})

You can also replace the rules or the data and run again.

Constructor arguments

All arguments are optional.

Caution: Do not set rules and rules_file together. Do not set data and data_file together. If you do, Rumo raises a ValueError.

The next sections give more information about each step.

Load rules

A Rumo object keeps one rule program. When you load a new rule program, it replaces the old rule program.

Load rules from a string

rumo.read_rules_str("""
@prefix : <http://example.org/> .
adult(?name) :- data(?p, :name, ?name), data(?p, :age, ?age), ?age >= 18 .
""")

You can also use the rules argument of the constructor.

Load rules from a file

rumo.read_rules_file("examples/sample.rls")

You can also use the rules_file argument of the constructor.

Rumo reads the file immediately. If you change the file later, load it again.

Paths in @import statements

A rule program can read files with @import statements. Rumo finds these files in this sequence:

1. The data resources of the Rumo object. Refer to Load data.
2. Files relative to a directory:
   • For a rule file, this is the directory of the rule file.
   • For a rule string, this is the current working directory.

Remove the rules

rumo.reset_rules()

If you run a Rumo object without rules, it raises a ValueError.

Errors in the rules

If the rule program has an error, run raises a RuntimeError. The message shows the line and the column of the error:

[01] Error: expected `)`
   ╭─[ :1:8 ]
   │
 1 │ foo(?x :- .
   │        │
   │        ╰─ expected `)`
───╯

Load data

Rumo keeps the data in memory. Rumo does not write temporary files.

Each data item has a resource name. The rule program uses this name to read the data.

Load a DataFrame

rumo.read_dataframe(df)

The DataFrame can be a pandas DataFrame or a Polars DataFrame. Rumo converts the DataFrame to Turtle. Refer to DataFrame to RDF conversion.

Note: For a pandas DataFrame, Rumo uses Polars to convert the data. If pyarrow is available, Rumo uses it. If pyarrow is not available, Rumo uses Python lists.

Load data from a string

rumo.read_data_str("""
@prefix : <http://example.org/> .
:a :name "Alice" .
:b :name "Bob" .
""", predicate="data")

Load data from a file

rumo.read_data_file("examples/sample.ttl", predicate="data")

Rumo reads the file immediately.

If you do not set format, Rumo uses the file extension:

Connect the data to the rules

There are two methods to connect the data to the rule program.

Method 1: Set a predicate

Set the predicate argument. Rumo then adds an @import statement to the rule program. For example, predicate="data" adds this statement:

@import data :- turtle { resource = "data.ttl" } .

For RDF data, the predicate has three columns: subject, property and value. Use it in the rules like this:

@prefix : <http://example.org/> .
named(?row, ?name) :- data(?row, :name, ?name) .

Use this method when you write the rules for Rumo.

Method 2: Use the resource name in the rules

Do not set the predicate argument. Set the resource argument to the name in the @import statement of the rules.

For example, examples/sample.rls has this statement:

@import r :- turtle { resource = "sample.ttl" } .

To run this file on a DataFrame, set resource="sample.ttl":

rumo = Rumo(rules_file="examples/sample.rls", data=df, resource="sample.ttl")

Rumo gives the DataFrame to the rules. Rumo does not read the file sample.ttl.

Use this method when you have rule files that already have @import statements.

Load more than one data item

You can load more than one data item. Give each data item a different resource name:

rumo.read_dataframe(people, resource="people.ttl", predicate="person")
rumo.read_dataframe(orders, resource="orders.ttl", predicate="order", row_stem="o")

Caution: If you load data with a resource name that already exists, the new data replaces the old data.

Caution: Two DataFrames with the same base_url and row_stem give the same row names (:r0, :r1, …). To keep the rows different, set a different row_stem for each DataFrame.

See the loaded data

The resources property gives the resource names:

print(rumo.resources)   # ['people.ttl', 'orders.ttl']

Remove the data

rumo.reset_data()

Run the rules

Use the run method:

results = rumo.run(params={"GOOD_SCORE": 90}, predicates=["good"])

run returns a RumoResults object. Refer to Use the results.

Parameters

A rule program can use global parameters, for example $GOOD_SCORE. Give the values in the params dictionary. Do not write the $ character in the key.

rumo.run(params={"GOOD_SCORE": 90, "CITY": "Oviedo"})

Rumo converts each Python value to a nemo value:

Other Python types cause a ValueError.

Select the predicates

If you do not set predicates, run returns:

1. The predicates in @output statements and @export statements of the rules.
2. If the rules have no such statements, all predicates that the rules calculate.

To get a different set of predicates, set predicates:

results = rumo.run(predicates=["good", "data"])

You can select any predicate in the rule program, also an imported predicate.

If a predicate does not exist, run raises a RuntimeError.

Note: run does not write the files of @export statements. run only returns the facts.

Errors

Use the results

run returns a RumoResults object. This object has one table for each predicate.

Column names

Rumo uses the variable names in the heads of the rules as column names. For example, this rule gives the columns name and score:

good(?name, ?score) :- ...

If a column has no variable name, Rumo uses col0, col1, and so on. This occurs for imported predicates and for constants in rule heads.

If more than one rule calculates the same predicate, Rumo uses the first available name for each column.

To set different column names, use the columns argument:

results.to_pandas("good", columns=["person", "points"])

The number of names must be the same as the number of columns. If it is not, Rumo raises a ValueError.

Get a pandas DataFrame

df = results.to_pandas("good")

This method needs pandas.

Get a Polars DataFrame

df = results.to_polars("good")

Get a list of dictionaries

rows = results.to_dicts("good")
# [{'name': 'Bob', 'score': 92.0}, {'name': 'Dave', 'score': 95.1}]

Get a list of tuples

rows = results.rows("good")
# [('Bob', 92.0), ('Dave', 95.1)]

Get all the results

to_dict returns a dictionary with one list of dictionaries for each predicate:

results.to_dict()
# {'good': [{'name': 'Bob', 'score': 92.0}, ...]}

Omit the predicate name

If the results have only one predicate, you can omit the predicate name:

results.to_pandas()

If the results have more than one predicate, you must give the name. If you do not, Rumo raises a ValueError.

Use the results as a mapping

A RumoResults object operates like a read-only dictionary:

results.predicates()     # ['good']
results.columns("good")  # ['name', 'score']
"good" in results        # True
len(results)             # 1
for predicate in results:
    print(predicate, results[predicate])

results["good"] returns a list of dictionaries. If the predicate does not exist, it raises a KeyError.

Value types

Rumo converts each nemo value to a Python value:

Note: Turtle reads numbers such as 88.5 as xsd:decimal. Thus, numbers from a DataFrame become float values in the results.

DataFrame to RDF conversion

rumo converts a DataFrame to RDF in the Turtle format. The command-line tool, pyrumo.to_turtle and Rumo use the same conversion.

Conversion rules

• Each row becomes a subject. The name of the subject is <base_url><row_stem><index>. The first index is 0.
• Each column becomes a property. The name of the property is <base_url><column name>.
• Each cell becomes one triple: the row, the column property and the value.

Values

Example

This table:

gives this Turtle text with base_url="http://example.org/" and row_stem="r":

prefix : <http://example.org/>

:r0 :name "Alice" ;
    :age 25 ;
    :score 88.5 .
:r1 :name "Bob" ;
    :age 30 ;
    :score 92.0 .

In a rule program, these triples have this form:

data(<http://example.org/r0>, <http://example.org/name>, "Alice")

Use a @prefix statement in the rules to write the IRIs in a short form:

@prefix : <http://example.org/> .
young(?name) :- data(?r, :name, ?name), data(?r, :age, ?age), ?age < 28 .

Limitations

Refer to Limitations and troubleshooting.

Python API reference

Module functions

Class Rumo

Rumo(rules=None, rules_file=None, data=None, data_file=None, *,
     format=None, resource=None, predicate=None,
     base_url="http://example.org/", row_stem="r")

Methods

Properties

Default resource names

Class RumoResults

In this table, predicate is optional if the results have only one predicate.

Exceptions

Limitations and troubleshooting

Limitations

DataFrame conversion

The DataFrame conversion has these limitations:

• rumo does not escape quotes or backslashes in strings. A string with a " character gives Turtle that is not correct.
• rumo writes column names directly as property names. A column name with a space or a special character gives Turtle that is not correct.
• rumo writes a null value as an empty string "". Thus, a rule finds a value for each cell, also for a cell with no data.

To prevent these problems, clean the DataFrame before you load it. For example, rename the columns and remove the null values.

Other limitations

• Rumo.run keeps the Python global interpreter lock (GIL) during the run. Other Python threads stop until the run is complete.
• Rumo.run does not write the files of @export statements. To write these files, use the command-line tool.
• The describe, print_info and to_turtle functions accept only Polars DataFrames.

Troubleshooting

Development

Repository structure

Run the Rust tests

cargo test
cargo test --features rules

Build and test the Python package

1. Make a virtual environment:

   python3 -m venv .venv
   source .venv/bin/activate
   

2. Install maturin:

   pip install maturin
   

3. Build and install the package:

   maturin develop --extras test
   

4. Run the tests:

   pytest
   

Build this manual

The manual uses mdBook.

1. Install mdBook:

   cargo install mdbook
   

2. Build the manual:

   mdbook build docs
   

   The HTML files are in docs/book.

3. To see the manual while you edit it, run:

   mdbook serve docs --open
   

Publish this manual

The Docs workflow (.github/workflows/docs.yml) publishes the manual to GitHub Pages.

Before the first publication, do this procedure one time:

1. On GitHub, open the repository settings.
2. Select Pages.
3. In Build and deployment, set Source to GitHub Actions.

The workflow starts in these conditions:

The manual is then at https://<owner>.github.io/<repository>/.

Writing style

Write the documentation in Simplified Technical English (ASD-STE100):

• Use short sentences. Write a maximum of 20 words in a procedure and 25 words in a description.
• Write one instruction in each sentence.
• Use the active voice.
• Use the imperative form in procedures.
• Use the same word for the same thing.

Make a release

1. Change version in Cargo.toml. This is also the version of the Python package.
2. Run cargo check to update Cargo.lock.
3. Commit and push these changes.
4. Publish a GitHub release. The tag must be the version with a v before it, for example v0.2.0.

The Binaries workflow attaches the command-line archives to the release. The Python workflow publishes the pyrumo wheels to PyPI.

Each workflow first compares the tag with the version in Cargo.toml. If they are different, the workflow stops before it builds.

If the check fails, do these steps:

1. Delete the GitHub release.
2. Delete the tag: git push --delete origin v0.2.0.
3. Change the version in Cargo.toml, then commit and push.
4. Publish the release again.