BEDOPS: the fast, highly scalable and easily-parallelizable genome analysis toolkit

BEDOPS is an open-source command-line toolkit that performs highly efficient and scalable Boolean and other set operations, statistical calculations, archiving, conversion and other management of genomic data of arbitrary scale. Tasks can be easily split by chromosome for distributing whole-genome analyses across a computational cluster.

You can read more about BEDOPS and how it can be useful for your research in the Overview documentation, as well as in the original manuscript.

• x86-64 (64-bit) binaries
• Installation instructions for Linux hosts

• Intel (64-bit, 10.10-10.15) installer package
• Installation instructions for Mac OS X hosts

• Source code (tar.gz)
• Source code (zip)
• Compilation instructions

• bedops - apply set operations on any number of BED inputs
• bedextract - efficiently extract BED features
• closest-features - matches nearest features between BED files

• bedmap - map overlapping BED elements onto target regions and optionally compute any number of common statistical operations

• sort-bed - apply lexicographical sort to BED data
• starch and unstarch - compress and extract BED data
• starchcat - merge compressed archives
• starchstrip - filter archive by chromosomes
• Conversion tools - convert common genomic formats to BED

• Parallel bam2bed and bam2starch - parallelized conversion and compression of BAM data
• Set operations with bedops
• Compression characteristics of starch
• Independent testing

• Table summary of BEDOPS toolkit
• Starch v2.2 format specification
• About nested elements
• Revision history
• Github release instructions
• Github repository

• How to install BEDOPS
• Usage examples of BEDOPS tools in action
• BEDOPS user forum
• BEDOPS discusssion mailing list

Citation

If you use BEDOPS in your research, please cite the following manuscript:

Shane Neph, M. Scott Kuehn, Alex P. Reynolds, et al. BEDOPS: high-performance genomic feature operations. Bioinformatics (2012) 28 (14): 1919-1920. doi: 10.1093/bioinformatics/bts277

Contents

• 1. Overview
  • 1.1. About BEDOPS
  • 1.2. Why you should use BEDOPS
    • 1.2.1. BEDOPS tools are flexible
    • 1.2.2. BEDOPS tools are fast and efficient
    • 1.2.3. BEDOPS tools make your work embarrassingly easy to parallelize
    • 1.2.4. BEDOPS tools are open, documented and supported
• 2. Installation
  • 2.1. Via pre-built packages
    • 2.1.1. Linux
    • 2.1.2. Mac OS X
  • 2.2. Via source code
    • 2.2.1. Linux
    • 2.2.2. Mac OS X
      • 2.2.2.1. Manual compilation
      • 2.2.2.2. Installation via Bioconda
      • 2.2.2.3. Installation via Homebrew
    • 2.2.3. Docker
    • 2.2.4. Cygwin
  • 2.3. Building an OS X installer package for redistribution
• 3. Revision history
  • 3.1. Current version
    • 3.1.1. v2.4.41
  • 3.2. Previous versions
    • 3.2.1. v2.4.40
    • 3.2.2. v2.4.39
    • 3.2.3. v2.4.38
    • 3.2.4. v2.4.37
    • 3.2.5. v2.4.36
    • 3.2.6. v2.4.35
    • 3.2.7. v2.4.34
    • 3.2.8. v2.4.33
    • 3.2.9. v2.4.32
    • 3.2.10. v2.4.31
    • 3.2.11. v2.4.30
    • 3.2.12. v2.4.29
    • 3.2.13. v2.4.28
    • 3.2.14. v2.4.27
    • 3.2.15. v2.4.26
    • 3.2.16. v2.4.25
    • 3.2.17. v2.4.24
    • 3.2.18. v2.4.23
    • 3.2.19. v2.4.22
    • 3.2.20. v2.4.21
    • 3.2.21. v2.4.20
    • 3.2.22. v2.4.19
    • 3.2.23. v2.4.18
    • 3.2.24. v2.4.17
    • 3.2.25. v2.4.16
    • 3.2.26. v2.4.15
    • 3.2.27. v2.4.14
    • 3.2.28. v2.4.13
    • 3.2.29. v2.4.12
    • 3.2.30. v2.4.11
    • 3.2.31. v2.4.10
    • 3.2.32. v2.4.9
    • 3.2.33. v2.4.8
    • 3.2.34. v2.4.7
    • 3.2.35. v2.4.6
    • 3.2.36. v2.4.5
    • 3.2.37. v2.4.4
    • 3.2.38. v2.4.3
    • 3.2.39. v2.4.2
    • 3.2.40. v2.4.1
    • 3.2.41. v2.4.0
    • 3.2.42. v2.3.0
    • 3.2.43. v2.2.0b
    • 3.2.44. v2.2.0
    • 3.2.45. v2.1.1
    • 3.2.46. v2.1.0
    • 3.2.47. v2.0.0b
    • 3.2.48. v2.0.0a
    • 3.2.49. v1.2.5b
    • 3.2.50. v1.2.5
    • 3.2.51. v1.2.3
• 4. Usage examples
  • 4.1. Visualizing the relationship of SNPs and generic genomic features
    • 4.1.1. BEDOPS tools in use
    • 4.1.2. Script
    • 4.1.3. Discussion
    • 4.1.4. Downloads
  • 4.2. Collapsing multiple BED files into a master list by signal
    • 4.2.1. BEDOPS tools in use
    • 4.2.2. Script
    • 4.2.3. Discussion
  • 4.3. Measuring the frequency of signed distances between SNPs and nearest DHSes
    • 4.3.1. BEDOPS tools in use
    • 4.3.2. Script
    • 4.3.3. Discussion
    • 4.3.4. Downloads
  • 4.4. Finding the subset of SNPs within DHSes
    • 4.4.1. BEDOPS tools in use
    • 4.4.2. Script
    • 4.4.3. Discussion
    • 4.4.4. Downloads
  • 4.5. Smoothing raw tag count data across the genome
    • 4.5.1. BEDOPS tools in use
    • 4.5.2. Script
  • 4.6. Efficiently creating Starch-formatted archives with a cluster
    • 4.6.1. BEDOPS tools in use
    • 4.6.2. Script
    • 4.6.3. Discussion
      • 4.6.3.1. Splitting BED files
      • 4.6.3.2. Compressing BED subsets
      • 4.6.3.3. Stitching together compressed sets
  • 4.7. Working with many input files at once with bedops and bedmap
    • 4.7.1. Discussion
• 5. Performance
  • 5.1. Test environment and data
  • 5.2. Set operations with bedops
    • 5.2.1. Direct merge (sorted)
    • 5.2.2. Complement and intersection
    • 5.2.3. Direct merge (unsorted)
    • 5.2.4. Discussion
  • 5.3. Compression characteristics of starch
    • 5.3.1. Compression efficiency
    • 5.3.2. Extraction time
  • 5.4. Independent testing
    • 5.4.1. Genomic Region Operation Kit (GROK)
  • 5.5. Worst-case memory performance
• 6. Reference
  • 6.1. Set operations
    • 6.1.1. bedops
      • 6.1.1.1. Inputs and outputs
        • 6.1.1.1.1. Input
        • 6.1.1.1.2. Output
      • 6.1.1.2. Usage
      • 6.1.1.3. Operations
        • 6.1.1.3.1. Everything (-u, –everything)
        • 6.1.1.3.2. Element-of (-e, –element-of)
        • 6.1.1.3.3. Not-element-of (-n, –not-element-of)
        • 6.1.1.3.4. Complement (-c, –complement)
        • 6.1.1.3.5. Difference (-d, –difference)
        • 6.1.1.3.6. Symmetric difference (-s, –symmdiff)
        • 6.1.1.3.7. Intersect (-i, –intersect)
        • 6.1.1.3.8. Merge (-m, –merge)
        • 6.1.1.3.9. Partition (-p, –partition)
        • 6.1.1.3.10. Chop (-w, –chop)
        • 6.1.1.3.11. Stagger (–stagger)
        • 6.1.1.3.12. Exclude (-x)
        • 6.1.1.3.13. Per-chromosome operations (–chrom)
        • 6.1.1.3.14. Range (–range)
      • 6.1.1.4. Starch support
      • 6.1.1.5. Error checking (–ec)
      • 6.1.1.6. Tips
        • 6.1.1.6.1. Chaining operations
        • 6.1.1.6.2. Sorting inputs
    • 6.1.2. bedextract
      • 6.1.2.1. How it works
      • 6.1.2.2. Inputs and outputs
        • 6.1.2.2.1. Input
        • 6.1.2.2.2. Output
      • 6.1.2.3. Usage
        • 6.1.2.3.1. Listing chromosomes
        • 6.1.2.3.2. Retrieving elements from a specific chromosome
        • 6.1.2.3.3. Retrieving elements which overlap target elements
          • 6.1.2.3.3.1. What are nested elements?
          • 6.1.2.3.3.2. Demonstration
      • 6.1.2.4. Downloads
    • 6.1.3. closest-features
      • 6.1.3.1. Inputs and outputs
        • 6.1.3.1.1. Input
        • 6.1.3.1.2. Output
      • 6.1.3.2. Usage
      • 6.1.3.3. Per-chromosome operations (–chrom)
      • 6.1.3.4. Error checking
      • 6.1.3.5. Downloads
    • 6.1.4. Nested elements
      • 6.1.4.1. Definition
      • 6.1.4.2. Example
      • 6.1.4.3. Why nested elements matter
  • 6.2. Statistics
    • 6.2.1. bedmap
      • 6.2.1.1. Inputs and outputs
        • 6.2.1.1.1. Input
        • 6.2.1.1.2. Output
      • 6.2.1.2. Usage
      • 6.2.1.3. Operations
        • 6.2.1.3.1. Overlap criteria
          • 6.2.1.3.1.1. Using --faster with --bp-ovr, --fraction-both, --exact or --range
        • 6.2.1.3.2. Score operations
        • 6.2.1.3.3. Non-score operations
          • 6.2.1.3.3.1. Echo
          • 6.2.1.3.3.2. Element and overlap statistics
          • 6.2.1.3.3.3. Indicator
        • 6.2.1.3.4. Modifiers
          • 6.2.1.3.4.1. Range
          • 6.2.1.3.4.2. Using --faster with --range
          • 6.2.1.3.4.3. Formatting score output
          • 6.2.1.3.4.4. Delimiters
      • 6.2.1.4. I/O event handling
      • 6.2.1.5. Per-chromosome operations (–chrom)
      • 6.2.1.6. Starch support
      • 6.2.1.7. Error checking
      • 6.2.1.8. Endlines
      • 6.2.1.9. Downloads
  • 6.3. File management
    • 6.3.1. Sorting
      • 6.3.1.1. sort-bed
        • 6.3.1.1.1. Migrating older BED and Starch files
        • 6.3.1.1.2. Inputs and outputs
          • 6.3.1.1.2.1. Input
          • 6.3.1.1.2.2. Output
        • 6.3.1.1.3. Usage
    • 6.3.2. Compression
      • 6.3.2.1. starch
        • 6.3.2.1.1. Inputs and outputs
          • 6.3.2.1.1.1. Input
          • 6.3.2.1.1.2. Output
        • 6.3.2.1.2. Requirements
        • 6.3.2.1.3. Usage
        • 6.3.2.1.4. Options
          • 6.3.2.1.4.1. Backend compression type
          • 6.3.2.1.4.2. Note
          • 6.3.2.1.4.3. Per-chromosome data integrity signature
          • 6.3.2.1.4.4. Compression progress
          • 6.3.2.1.4.5. Headers
          • 6.3.2.1.4.6. Unique tag
        • 6.3.2.1.5. Example
      • 6.3.2.2. unstarch
        • 6.3.2.2.1. Inputs and outputs
          • 6.3.2.2.1.1. Input
          • 6.3.2.2.1.2. Output
        • 6.3.2.2.2. Requirements
        • 6.3.2.2.3. Usage
          • 6.3.2.2.3.1. Extraction
          • 6.3.2.2.3.2. Archive attributes
            • 6.3.2.2.3.2.1. Metadata
            • 6.3.2.2.3.2.2. Note
            • 6.3.2.2.3.2.3. Timestamp
            • 6.3.2.2.3.2.4. Compression type
            • 6.3.2.2.3.2.5. Version
          • 6.3.2.2.3.3. Whole-file or per-chromosome attributes
            • 6.3.2.2.3.3.1. Data integrity
            • 6.3.2.2.3.3.2. Elements
            • 6.3.2.2.3.3.3. Bases
            • 6.3.2.2.3.3.4. Duplicate element(s)
            • 6.3.2.2.3.3.5. Nested element(s)
        • 6.3.2.2.4. Example
      • 6.3.2.3. starchcat
        • 6.3.2.3.1. Parallelization
        • 6.3.2.3.2. Inputs and outputs
          • 6.3.2.3.2.1. Input
          • 6.3.2.3.2.2. Output
        • 6.3.2.3.3. Usage
          • 6.3.2.3.3.1. Per-chromosome data integrity signature
          • 6.3.2.3.3.2. Example
      • 6.3.2.4. Starch (v2.x) specification
        • 6.3.2.4.1. Archive structure
        • 6.3.2.4.2. Magic bytes
        • 6.3.2.4.3. Chromosome streams
        • 6.3.2.4.4. Metadata
          • 6.3.2.4.4.1. Data
            • 6.3.2.4.4.1.1. Archive
            • 6.3.2.4.4.1.2. Streams
          • 6.3.2.4.4.2. Offset
          • 6.3.2.4.4.3. Hash
        • 6.3.2.4.5. Padding
      • 6.3.2.5. starch-diff
        • 6.3.2.5.1. Inputs and outputs
          • 6.3.2.5.1.1. Input
          • 6.3.2.5.1.2. Output
        • 6.3.2.5.2. Requirements
        • 6.3.2.5.3. Usage
    • 6.3.3. Data conversion
      • 6.3.3.1. convert2bed
        • 6.3.3.1.1. Dependencies
        • 6.3.3.1.2. Source
        • 6.3.3.1.3. Usage
        • 6.3.3.1.4. Example
      • 6.3.3.2. bam2bed
        • 6.3.3.2.1. Dependencies
        • 6.3.3.2.2. Source
        • 6.3.3.2.3. Usage
        • 6.3.3.2.4. Example
        • 6.3.3.2.5. Column mapping
        • 6.3.3.2.6. Downloads
      • 6.3.3.3. Parallel bam2bed
        • 6.3.3.3.1. Dependencies
        • 6.3.3.3.2. Source
        • 6.3.3.3.3. Usage
      • 6.3.3.4. Parallel bam2starch
        • 6.3.3.4.1. Dependencies
        • 6.3.3.4.2. Source
        • 6.3.3.4.3. Usage
      • 6.3.3.5. gff2bed
        • 6.3.3.5.1. Dependencies
        • 6.3.3.5.2. Source
        • 6.3.3.5.3. Usage
        • 6.3.3.5.4. Example
        • 6.3.3.5.5. Column mapping
        • 6.3.3.5.6. Downloads
      • 6.3.3.6. gtf2bed
        • 6.3.3.6.1. Dependencies
        • 6.3.3.6.2. Source
        • 6.3.3.6.3. Usage
        • 6.3.3.6.4. Example
        • 6.3.3.6.5. Column mapping
        • 6.3.3.6.6. Downloads
      • 6.3.3.7. gvf2bed
        • 6.3.3.7.1. Dependencies
        • 6.3.3.7.2. Source
        • 6.3.3.7.3. Usage
        • 6.3.3.7.4. Example
        • 6.3.3.7.5. Column mapping
        • 6.3.3.7.6. Downloads
      • 6.3.3.8. psl2bed
        • 6.3.3.8.1. Dependencies
        • 6.3.3.8.2. Source
        • 6.3.3.8.3. Usage
        • 6.3.3.8.4. Example
        • 6.3.3.8.5. Column mapping
        • 6.3.3.8.6. Downloads
      • 6.3.3.9. rmsk2bed
        • 6.3.3.9.1. Dependencies
        • 6.3.3.9.2. Source
        • 6.3.3.9.3. Usage
        • 6.3.3.9.4. Example
        • 6.3.3.9.5. Column mapping
        • 6.3.3.9.6. Downloads
      • 6.3.3.10. sam2bed
        • 6.3.3.10.1. Dependencies
        • 6.3.3.10.2. Source
        • 6.3.3.10.3. Usage
        • 6.3.3.10.4. Example
        • 6.3.3.10.5. Column mapping
        • 6.3.3.10.6. Downloads
      • 6.3.3.11. vcf2bed
        • 6.3.3.11.1. Dependencies
        • 6.3.3.11.2. Source
        • 6.3.3.11.3. Usage
        • 6.3.3.11.4. Customized variant handling
        • 6.3.3.11.5. Example
        • 6.3.3.11.6. Column mapping
        • 6.3.3.11.7. Downloads
      • 6.3.3.12. wig2bed
        • 6.3.3.12.1. Source
        • 6.3.3.12.2. Usage
        • 6.3.3.12.3. Example
        • 6.3.3.12.4. Downloads
• 7. Summary
  • 7.1. Set operation and statistical utilities
    • 7.1.1. bedextract
    • 7.1.2. bedmap
    • 7.1.3. bedops
    • 7.1.4. closest-features
  • 7.2. Sorting
    • 7.2.1. sort-bed
  • 7.3. Compression and extraction
    • 7.3.1. starch
    • 7.3.2. unstarch
    • 7.3.3. starchcat
    • 7.3.4. starchstrip
• 8. Release
  • 8.1. Preparation
  • 8.2. Release
  • 8.3. Celebrate
• 9. Placeholder