Metadata-Version: 2.1
Name: pfio
Version: 1.5.0
Summary: PFN IO library
Home-page: http://github.com/pfnet/pfio
Author: Tianqi Xu, Kota Uenishi
Author-email: tianqi@preferred.jp, kota@preferred.jp
License: UNKNOWN
Keywords: filesystem hdfs chainer development
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Build Tools
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: System :: Filesystems
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Provides-Extra: test
Provides-Extra: doc
License-File: LICENSE

## PFIO

PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports

- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.


## Dependency

- HDFS client and libhdfs for HDFS access
- CPython >= 3.6

## Installation and Document build

Installation

```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```

Documentation
```sh
$ cd pfio/docs
$ make html
$ open build/html/index.html
```

Test
```sh
$ cd pfio
$ pip install .[test]
$ pytest tests/
```

## How to use

Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.

## Release

Check [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.

Run tests locally:

```sh
$ pip install --user -e .[test]
$ pytest
```

Bump version numbers in `pfio/version.py` and in `docs/source/conf.py` .

Push and open a pull request to invoke CI. Once CI passed and the pull request merged,
tag a release:

```sh
$ git tag -s X.Y.Z
$ git push --tags
```

Build:

```sh
$ rm -rf dist
$ pip3 install --user build
$ python3 -m build
```

Release to PyPI:

```sh
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
```


