Metadata-Version: 2.1
Name: redast
Version: 0.0.4
Summary: UNKNOWN
Home-page: https://github.com/rilshok/redast
Author: Vladislav A. Proskurov
Author-email: rilshok@pm.me
License: MIT
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3.9
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 1 - Planning
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE

# REDAST - Remote Data Storage

Redast provides a flexible interface for storing data in different types of storages.
The organization of data storage is carried out according to the principle of a data lake.

## Installation

```bash
pip install redast
```

Or from GitHub

```bash
git clone git@github.com:rilshok/redast.git
cd redast
pip install -e .
```

or

```bash
pip install git+https://github.com/rilshok/redast.git
```

## Supported storage methods

### MemoryStorage

Local storage of data in RAM

```python
from redast import MemoryStorage

storage = MemoryStorage()
```

### DriveStorage

Local storage of data on drive. To open a storage, you must specify the `root` folder.
This call will create an empty directory `./myStorage`.

```python
from redast import DriveStorage

storage = DriveStorage(root="myStorage", create=True)
```

## Data storage interface

## Push data into storage

```python
key = storage.push(b"hello world")
print(key)
```

```plain
021ced8799296ceca557832ab941a50b4a11f83478cf141f51f933f653ab9fbcc05a037cddbed06e309bf334942c4e58cdf1a46e237911ccd7fcf9787cbc7fd0
```

## Pull data from storage

```python
data = storage.load(key)
print(data)
```

```plain
b'hello world'
```

## Check for data in storage

```python
ok = storage.exists(key)
bad = storage.exists("brokenkey")
print(ok, bad)
```

```plain
True False
```

## Pop data from storage

```python
data = storage.pop(key)
exist = storage.exists(key)
print(data, exist)
```

```plain
b'hello world' False
```

## Custom links

The ``link`` method links the saved data to the user identifier.

```python
storage.link("hello").push(b"hello world")
data = storage.link("hello").load()
print(data)
```

```plain
b'hello world'
```

Any python object can act as an identifier, even a lambda function.

```python
key = lambda x: x**2
storage.link(key).push(b"squaring")
qrt = storage.link(key).load()
print(qrt)
```

```plain
b'squaring'
```

A sequence of objects can be used as an identifier.

```python
storage.link("hello", "world").push(b"hello world")
data = storage.link("hello", "world").pop()
print(data)
```

```plain
b'hello world'
```

## Data packaging

Some form of pre-processing may be required to store the data. For large storages, data compression may be appropriate. Public storage may require encryption. The REDAST supports different preprocessing.

### Compression

```python
data = b"hello"*5
key = storage.compression.push(data)
compressed = storage.load(key)
data = storage.compression.load(key)
print(len(compressed), len(data), compressed, data, sep="\n")
```

```plain
16
25
b'x\x9c\xcbH\xcd\xc9\xc9\xcf\xc0B\x00\x00\x86\xc4\ne'
b'hellohellohellohellohello'
```

### Base64

```python
key = storage.base64.push(b"hello world")
data_base64 = storage.load(key)
data = storage.base64.load(key)
print(data_base64, data, sep="\n")
```

```plain
b'aGVsbG8gd29ybGQ='
b'hello world'
```

### Pickling of python objects

Different types of pre-processing can be combined into a single conveyor line

```python
data = dict(a=1, b=2)
key = storage.pickling.push(data)
data = storage.pickling.load(key)
print(data)
```

```plain
{'a': 1, 'b': 2}
```

### Encryption

Encryption requires an encryption key or user password, from which the encryption key will be generated. When using encryption with a password, you must choose an arbitrary seed to create the salt.

You can generate a random key

```python
from redast.core import Encryption

secret = Encryption.generate_key()
print(secret)
```

```plain
pcu398oGPbEIoc8UgXGnmjJUZOId1bQRWJ5VmpH2gQo=
```

Generating an encryption key using a passphrase

```python
from redast.core import Encryption

secret1 = Encryption.generate_key(password="mypassword", seed=2022)
secret2 = Encryption.generate_key(password="mypassword", seed=2022)
secret3 = Encryption.generate_key(password="mypassword", seed=2021)
print(secret1, secret2, secret3, sep="\n")
```

```plain
DZ6xJLl9D7MkHkYt6JVef1j6o36KU7XzXzYRZrUDt0w=
DZ6xJLl9D7MkHkYt6JVef1j6o36KU7XzXzYRZrUDt0w=
I4sopjxisx6wb7dCkOtwoBxDvo5lnb6tctcpdpS5jGg=
```

Encryption with an encryption key

```python
from redast.core import Encryption

secret = Encryption.generate_key()
key = storage.encryption(key=secret).push(b"topsecret")

encrypted = storage.load(key)
decrypted = storage.encryption(key=secret).load(key)

print(encrypted, decrypted, sep="\n")
```

```plain
b"\x00k\xd0=\x81\xedh\x94x\xed\xb0\x14\xe3\r\x96'"
b'topsecret'
```

Encryption with a password

```python
key = storage.encryption(password="mypassword", seed=10).push(b"topsecret")

encrypted = storage.load(key)
decrypted = storage.encryption(password="mypassword", seed=10).load(key)

print(encrypted, decrypted, sep="\n")
```

```plain
b'\xe4\xc6\x0bc\xd0\x92\xcb\xaeQ\x0ey&\x83\xb9\x9d@'
b'topsecret'
```

### Pipeline data packing

In this example, the data will be converted in the following order before being stored:

* First, the object to be saved will be pickled
* After pickling, the result will be compressed
* The compression result is then encrypted
* The result of encryption is converted to base64
* The base 64 string is pushed into storage

```python
data = [("hello", "world"), "foo"]

auth = dict(
    password="mypassword",
    seed=2022,
)
pipe = storage.base64.encryption(**auth).compression(level=9).pickling

key = pipe.push(data)
stored = storage.load(key)
data = pipe.load(key)

print(stored, data, sep="\n")
```

```plain
b'z2bqXGPZoXkN_-3LB9ocrBeWi4rax7le4wJncKe1Cnz7ep-KoHgFDeV2DisMq6Az'
[('hello', 'world'), 'foo']
```

The default packing options for pipeline data can be set when creating the storage itself

```python
from redast import MemoryStorage

data = [("hello", "world"), "foo"]

storage = MemoryStorage(
    compression = 9,
    encryption_password = "mypassword",
    encryption_seed = 2022,
)
pipe = storage.base64.encryption.compression.pickling


key = pipe.push(data)
stored = storage.load(key)
data = pipe.load(key)

print(stored, data, sep="\n")
```

```bash
b'z2bqXGPZoXkN_-3LB9ocrBeWi4rax7le4wJncKe1Cnz7ep-KoHgFDeV2DisMq6Az'
[('hello', 'world'), 'foo']
```

### Pipeline data packing with custom links

```python
from redast import MemoryStorage

storage = MemoryStorage()
pipe = storage.base64.compression

data = b"hello world"
pipe.link("mylink").push(data)
data = pipe.link("mylink").load()

print(data)
```

```plain
b'hello world'
```


