README
¶
1 项目介绍
1.1简介
CangHaiOS是分布式对象存储产品，可部署在X86通用服务器上，采用分布式架构，提供企业级高可用和线性扩展能力，广泛适用于金融、运营商、政府公共事业等行业云存储场景。
1.2 整体架构
1.2.1主要模块说明
- Filer：OSS集群的请求入口，除接收http请求外，还有ftp、sftp请求。
- Master：集群状态管理服务集群，负责卷服务集群与映射服务集群的状态管理。
- Partition：文件元数据持久化服务器集群，使用Mysql数据库，负责路径与对象ID等元数据映射关系存储。
- Volume：文件数据持久化集群，集群可采用双副本、三副本保证高可靠性。
- BinlogServer：记录文件操作日志。
- Backup：根据文件操作日志，在多数据中心同步文件进行操作。
- Fuse：在Linux系统中，将对象存储的存储空间(Bucket)挂载到本地文件系统中，能够像操作本地文件一样操作文件，实现数据共享。
1.3 产品亮点
1.3.1 高可用性、可靠性、扩展性
- 每个文件的不同冗余存储在同一个区域内多个设备上，确保数据持久性和可用性。
- 存储空间大小无限制，可以根据所需存储量无限扩展存储空间，解决了传统硬件存储扩容问题。
- 当检测到数据有部分损坏或丢失时，OSS会利用冗余数据重建并修复损坏数据。
1.3.2 方便快捷的使用方式
- 提供标准的RESTful API接口、丰富的SDK包、客户端工具、控制台。可以像使用文件一样方便地上传、下载、检索、管理海量数据。
- 兼容Amazon S3接口，可轻松实现亚马逊、阿里云等公有云数据与CangHaiOS之间的数据迁移。
- 提供标准HDFS接口，大数据应用场景中可直接将数据存在对象存储上。
- 支持文件生命周期管理。可以通过设置生命周期规则，将到期数据批量删除或者转储为更低成本的低频访问存储或归档存储。
1.3.3 强大、灵活的安全机制
提供企业级多层次安全防护，包括防盗链、服务端加密、客户端加密、合规保留策略、版本控制等。
1.3.4 QoS控制
- 可针对单个租户进行tps控制、流量控制、图片请求处理等控制。
1.3.5 提供丰富、强大的增值业务
- 图片处理：支持JPG、PNG、BMP、WebP等多种图片格式的转换，以及缩略图、剪裁、水印、缩放等多种操作。
- 内容加速分发：OSS作为源站，搭配CDN进行内容分发，提升同一个文件被重复下载的体验。
1.4 产品功能特性
- 上传文件：创建存储空间后，可以上传任意文件到该存储空间
- 下载文件：当文件上传至存储空间后，可以将文件下载到本地
- 搜索文件：可以在存储空间中快速查找目标文件
- 删除文件：支持删除文件或者文件夹
- 数据迁移：支持
- 数据冗余保护：数据支持1~3副本存储，存储持久性可达99.99995%，支持纠删码。
- 数据加密：支持客户端和服务器端加密，可以选择合适的加密方式将数据加密后存储。
- 图片处理：对存储的jpg、png图片执行不同的图片处理操作，如格式转换、裁剪、缩放、旋转、水印等。
- 多种接入方式：http、ftp、sftp
1.5产品规格
- 系统架构：分布式架构
- 存储介质：HDD
- 网络类型：以太网
- 数据冗余保护：1~3副本
- 单集群内支持的节点数：1024
- 单集群支持最大用户个数：100000
- 单集群支持最大存储空间个数：1000000
- 系统支持最大对象个数：万亿
- 单用户支持最大存储桶数：100
- 单桶支持最大对象数：100亿
- 单个文件大小：6T
- 可用性：99.99%
2 构建fuse版本
操作系统：CentOS7.3 <br>
# sh make-CangHaiOs.sh
3 fuse启动
操作系统：CentOS7.3 <br>
/opt/ossfuse -v=0 -http=true -log_dir=/opt/log/ossfuse/ ossfuse -account=test -bucket=test -mountpoint=/mnt/ossfuse -ossendpoint=oss.testdomain.com -ip=10.1.1.1
4 License
Copyright 2021 Suning Inc
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Directories
¶
| Path | Synopsis |
|---|---|
|
CangHaiOs_git
|
|
|
src/code.google.com/p/go.image/bmp
Package bmp implements a BMP image decoder and encoder.
|
Package bmp implements a BMP image decoder and encoder. |
|
src/code.google.com/p/go.image/tiff
Package tiff implements a TIFF image decoder and encoder.
|
Package tiff implements a TIFF image decoder and encoder. |
|
src/code.google.com/p/go.image/tiff/lzw
Package lzw implements the Lempel-Ziv-Welch compressed data format, described in T. A. Welch, “A Technique for High-Performance Data Compression”, Computer, 17(6) (June 1984), pp 8-19.
|
Package lzw implements the Lempel-Ziv-Welch compressed data format, described in T. A. Welch, “A Technique for High-Performance Data Compression”, Computer, 17(6) (June 1984), pp 8-19. |
|
src/code.google.com/p/go.image/vp8
Package vp8 implements a decoder for the VP8 lossy image format.
|
Package vp8 implements a decoder for the VP8 lossy image format. |
|
src/code.google.com/p/go.image/vp8l
Package vp8l implements a decoder for the VP8L lossless image format.
|
Package vp8l implements a decoder for the VP8L lossless image format. |
|
src/code.google.com/p/go.image/webp
Package webp implements a decoder for WEBP images.
|
Package webp implements a decoder for WEBP images. |
|
src/code.google.com/p/goprotobuf/proto
Package proto converts data structures to and from the wire format of protocol buffers.
|
Package proto converts data structures to and from the wire format of protocol buffers. |
|
src/code.google.com/p/goprotobuf/protoc-gen-go/descriptor
Package google_protobuf is a generated protocol buffer package.
|
Package google_protobuf is a generated protocol buffer package. |
|
src/code.google.com/p/goprotobuf/protoc-gen-go/generator
The code generator for the plugin for the Google protocol buffer compiler.
|
The code generator for the plugin for the Google protocol buffer compiler. |
|
src/code.google.com/p/goprotobuf/protoc-gen-go/plugin
Package google_protobuf_compiler is a generated protocol buffer package.
|
Package google_protobuf_compiler is a generated protocol buffer package. |
|
src/code.google.com/p/snappy-go/snappy
Package snappy implements the snappy block-based compression format.
|
Package snappy implements the snappy block-based compression format. |
|
src/github.com/disintegration/imaging
Package imaging provides basic image manipulation functions (resize, rotate, flip, crop, etc.).
|
Package imaging provides basic image manipulation functions (resize, rotate, flip, crop, etc.). |
|
src/github.com/go-fuse/fuse/nodefs
The nodefs package offers a high level API that resembles the kernel's idea of what an FS looks like.
|
The nodefs package offers a high level API that resembles the kernel's idea of what an FS looks like. |
|
src/github.com/go-fuse/fuse/ossfs
ossfs 这个包 主要是从objectfs copy过来，然后进行调整性优化。
|
ossfs 这个包 主要是从objectfs copy过来，然后进行调整性优化。 |
|
src/github.com/go-fuse/fuse/test
Package test holds the tests for Go-FUSE and is not for end-user consumption.
|
Package test holds the tests for Go-FUSE and is not for end-user consumption. |
|
src/github.com/go-redis/redis
Package redis implements a Redis client.
|
Package redis implements a Redis client. |
|
src/github.com/go-redis/redis/internal/consistenthash
Package consistenthash provides an implementation of a ring hash.
|
Package consistenthash provides an implementation of a ring hash. |
|
src/github.com/go-sql-driver/mysql
Go MySQL Driver - A MySQL-Driver for Go's database/sql package
|
Go MySQL Driver - A MySQL-Driver for Go's database/sql package |
|
src/github.com/pkg/errors
Package errors provides simple error handling primitives.
|
Package errors provides simple error handling primitives. |
|
src/github.com/pkg/sftp
Package sftp implements the SSH File Transfer Protocol as described in https://filezilla-project.org/specs/draft-ietf-secsh-filexfer-02.txt
|
Package sftp implements the SSH File Transfer Protocol as described in https://filezilla-project.org/specs/draft-ietf-secsh-filexfer-02.txt |
|
src/github.com/resize
Package resize implements various image resizing methods.
|
Package resize implements various image resizing methods. |
|
src/github.com/rwcarlsen/goexif/exif
Package exif implements decoding of EXIF data as defined in the EXIF 2.2 specification (http://www.exif.org/Exif2-2.PDF).
|
Package exif implements decoding of EXIF data as defined in the EXIF 2.2 specification (http://www.exif.org/Exif2-2.PDF). |
|
src/github.com/rwcarlsen/goexif/mknote
Package mknote provides makernote parsers that can be used with goexif/exif.
|
Package mknote provides makernote parsers that can be used with goexif/exif. |
|
src/github.com/rwcarlsen/goexif/tiff
Package tiff implements TIFF decoding as defined in TIFF 6.0 specification at http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf
|
Package tiff implements TIFF decoding as defined in TIFF 6.0 specification at http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf |
|
src/github.com/syndtr/goleveldb/leveldb
Package leveldb provides implementation of LevelDB key/value database.
|
Package leveldb provides implementation of LevelDB key/value database. |
|
src/github.com/syndtr/goleveldb/leveldb/cache
Package cache provides interface and implementation of a cache algorithms.
|
Package cache provides interface and implementation of a cache algorithms. |
|
src/github.com/syndtr/goleveldb/leveldb/comparer
Package comparer provides interface and implementation for ordering sets of data.
|
Package comparer provides interface and implementation for ordering sets of data. |
|
src/github.com/syndtr/goleveldb/leveldb/filter
Package filter provides interface and implementation of probabilistic data structure.
|
Package filter provides interface and implementation of probabilistic data structure. |
|
src/github.com/syndtr/goleveldb/leveldb/iterator
Package iterator provides interface and implementation to traverse over contents of a database.
|
Package iterator provides interface and implementation to traverse over contents of a database. |
|
src/github.com/syndtr/goleveldb/leveldb/journal
Package journal reads and writes sequences of journals.
|
Package journal reads and writes sequences of journals. |
|
src/github.com/syndtr/goleveldb/leveldb/memdb
Package memdb provides in-memory key/value database implementation.
|
Package memdb provides in-memory key/value database implementation. |
|
src/github.com/syndtr/goleveldb/leveldb/opt
Package opt provides sets of options used by LevelDB.
|
Package opt provides sets of options used by LevelDB. |
|
src/github.com/syndtr/goleveldb/leveldb/storage
Package storage provides storage abstraction for LevelDB.
|
Package storage provides storage abstraction for LevelDB. |
|
src/github.com/syndtr/goleveldb/leveldb/table
Package table allows read and write sorted key/value.
|
Package table allows read and write sorted key/value. |
|
src/github.com/syndtr/goleveldb/leveldb/util
Package util provides utilities used throughout leveldb.
|
Package util provides utilities used throughout leveldb. |
|
src/github.com/tgulacsi/go-cdb
Package cdb reads and writes cdb ("constant database") files.
|
Package cdb reads and writes cdb ("constant database") files. |
Click to show internal directories.
Click to hide internal directories.