ForestClaw

A parallel, adaptive library for logically Cartesian, mapped, multiblock domains

What is ForestClaw?

ForestClaw is a parallel, multi-block adaptive finite volume library for solving PDEs on mapped, logically Cartesian meshes. For solving hyperbolic problems using explicit, single step algorithms, ForestClaw's block-structured adaptive algorithm, including multi-rate time stepping uses the Berger, Oliger and Colella AMR algorithms (JCP, 1984, 1989). The hyperbolic solvers are currently based on ClawPack (R. J. LeVeque). Future plans include support for general method-of-lines solvers in a multi-rate setting.

Where ForestClaw departs from the standard Berger-Oliger-Colella block-structured approach is that the multi-resolution grid hierarchy is not stored as overlapping, nested grids but rather as a composite structure of non-overlapping fixed sized grids, each of which is stored as a leaf in a forest of quad- or octrees. The particular code base we use for managing the tree is p4est (www.p4est.org), (Carsten Burstedde, Univ. of Bonn).

Back to the top

Why develop another AMR code?

Currently, there are several AMR codes for doing patch-structured or block-structured AMR. What are the advantages of the ForestClaw approach?

By using an external library for mesh management, we completely separate issues related to grid connectivity, load balancing, and regridding, from the underlying numerics. With p4est, we can exploit state-of-the-art technology for managing tree-based data structures, including the use of space filling curves for maintaining data locality in parallel.
Because of the composite quad- or octree layout, ghost cell exchanges occur in a regular pattern at patch boundaries, rather than between ghost cells on the fine grid and interior grid cells on the coarse grid. This eliminates the need to store the locations of where fine grids are positioned in the coarse grid. Also, only a limited number of neighbor configurations are allowed, again reducing the code complexity and simplifying the development of new numerical algorithms. This simplification is particularly advantageous in 3d.
p4est has been built for distributed parallelism and designed to allow transparent and efficient parallelization of numerical codes such as ForestClaw. For example, the p4est algorithms for regridding and re-balancing (ensuring that all levels are properly nested) has been shown to scale well on up to 262,144 CPU cores.
ForestClaw inherits the multi-block domain structure from p4est and so adaptive mesh refinement for non-rectangular domains, cubed-sphere domains, and other domains not easily handled by a single adaptive quadtree.

Back to the top

Progress on the ForestClaw project

(Fall, 2018) Preliminary work on CUDA support in ForestClaw. D. Calhoun leads team of four students to Lugano, Switzerland, where we participated in EuroHack 2018.
(Fall, 2018) Conservative corrections now included in ForestClaw
(Summer, 2018) ForestClaw receives continued NSF support with new three year grant!
(Spring 2018) D. Calhoun participated in Clawpack Hackathon (Nederland, CO)
(Fall 2017) Work on conservation fix. Stayed tuned!
(Summer 2017) ForestClaw is available in GitHub. See ForestClaw
(Spring 2017) Ash3d extension of ForestClaw (work with Melody Shih, and USGS (Cascade Volcanic Observatory))
(Winter, 2017) GeoClaw extension of ForestClaw largely complete. Begin work on 2.5 dimensional version.
(Fall 2016) Dynamic support for both Clawpack 4.x and Clawpack 5.0 libraries in all examples (advection, gas dynamics, acoustics, SWE, transport and so on).
(Summer 2016) Extend ForestClaw with GeoClaw (www.geoclaw.org).
(Spring 2016) Final progress on our paper; accuracy tests of ForestClaw.
(Fall/Winter, 2016) Developing performance testing tools for running on a local linux cluster (64-nodes) and JUQUEEN (Blue Gene/Q) machines; obtain scaling performance for a variety of test problems.
(Spring/Summer, 2015) Parallel multi-block code mostly working; parallel performance optimizations reduce regridding time significantly.
(Winter, 2015) Parallel IO for package and user options.
(Fall, 2014) Periodic domains
(Summer, 2014) Mapped, multiblock capabilities for several useful domains, including cubed disk and sphere,
(Spring, 2014) Build system using GNU Autotools
(Winter, 2014) Parco paper appears.
(Summer, 2013) Parallel (MPI) capabilities added; Parco paper submitted and accepted.
(Fall, 2012) General mappings available
(Fall, 2012) Multirate time stepping
(Summer, 2012) Development on ForestClaw begins
(March, 2012) D. Calhoun in charge of an AMR Working Group at HPC3, a regular workshop held at KAUST, Saudi Arabia.

Back to the top

Publications and Presentations

Recent talks and posters

SIAM Annual Meeting. July, 2018, Portland, OR
CSDMS Conference on Natural Hazards. May 21-May 29, 2018. Boulder, CO. (poster: .pdf)
SIAM Parallel Processing Conference, March 7-10, 2018. Tokyo, Japan. (talk: .pdf).
Chiba University, March 6, 2018. Tokyo, Japan. (talk: .pdf).
Pacific Northwest Chapter of SIAM/PNWNAS 2017, Oct. 27-29, 2017. Corvallis, OR.
University of Dusseldorf, Department of Mathematics, June 6, 2-17. (talk: .pdf).
Numerical methods for PDEs and their applications, May 29-June 2, 2017, Mittag-Leffler Institute, Stockholm, Sweden) (talk: .pdf)
KAUST workshop on predictive complex computational fluid dynamics, May 22-24, 2017 (KAUST, Saudi Arabia) (talk : Kaust.pdf)
Clifford Lectures, April 5-7, 2017, (New Orleans, LA) (talk : .pdf)
SIAM CSE, Feb. 27 - Mar. 3, 2017 (Atlanta, GA) (talk: .pdf, (poster: .pdf)
SIAM Parallel Processing, April 2016 (Paris, France) (talk 1: .pdf, (talk 2 : .pdf)
SIAM CSE, March, 2015 (Salt Lake City, Utah) (talk: .pdf, poster: .pdf)
HPC3, November, 2014 (Kaust). (.pdf)

Publications

D. Duplyakin, J. Brown, and D. Calhoun, Evaluating Active Learning with Cost and Memory Awareness, in 32nd IEEE International Parallel and Distributed Processing Symposium (IPDPS), Vancouver, WA, May 21-25 2018. (IEEE Xpress),
D. Calhoun, C. Burstedde, ForestClaw : A parallel algorithm for patch-based adaptive mesh refinement on a forest of quadtrees, (arXiv:1703.03116)
C. Burstedde, D. Calhoun, K. Mandli, A. Terrel, Hybrid forest-of-octrees AMR for hyperbolic conservation laws, (2014). (arXiv:1308.1472, iospress)

Back to the top

ForestClaw News

(June, 2018) We received 3 years of NSF funding to develop a fast solver for ForestClaw (Award #1819257, $315K). (project summary: .pdf)
(May, 2017) ForestClaw mentioned in SIAM News. (See AMR)
(April, 2017) R. J. LeVeque (developer of the original ClawPack-GeoClaw code) was the invited lecturer at the Tulane University Clifford Lecture Series. Several ClawPack/GeoClaw/ForestClaw developers were invited.
(February, 2017) At SIAM CSE 2017 (Atlanta, GA), the Clawpack-GeoClaw Mini-posterium won a poster prize, which came with $250, which will be given to junior members of the Clawpack group. Good work! (ForestClaw poster)
(November, 2016) First ForestClaw Hackathon. Kyle Mandli (Columbia Univ.) and Melody Shih (Columbia Univ.) spent four days in Boise, working on updates to ForestClaw. See the picture of our hike into the Boise hills here.
(September, 2016) D. Calhoun, ForestClaw developer, will attend the Clawpack Hackathon Hack-a-thon, in Seattle, WA, August 27-29, 2016, to work with Melody Shih (Columbia Univ.) on a ForestClaw implementation of GeoClaw
(April, 2016) Attend the three-part mini-symposium at the upcoming SIAM Parallel Processing Meeting (Paris, France, April 13-16, 2016) to find out more about progress in parallel adaptive methods, and to find out more about ForestClaw
(Wednesday 4/14 : MS18 - 10:25AM, MS26 - 2:40PM, MS34 - 4:50PM)
(February, 2015) Attend our minisymposium at SIAM CSE 2015 (Salt Lake City, March 14-18, 2015)
(Sunday 3/15 : MS102 - 9:10AM, MS127 - 1:30PM)
(September, 2014) Developer Donna Calhoun was recently featured in a Boise State update Math vs. the Volcano : Mitigating an Eruption's Aftermath
(July, 2014) We have recently received three years of National Science Foundation funding (Award #1419108, $194K) for development of ForestClaw. Please see the project_summary for a description of the award. Collaborators on this project include Carsten Burstedde (Univ. Bonn, Germany), D. Ketcheson (Kaust, Saudi Arabia) and D. George (USGS, Vancouver, WA).

Back to the top

Working on the ForestClaw project

There are funding opportunities for students interested in working on developing numerical solvers and applications using ForestClaw.

Currently, we have funding for a PhD position, staring Fall 2019. For more details on this position, see Funding Opportunity.

If you are a student interested in being involved the ForestClaw project, please email Donna Calhoun at donnacalhoun@boisestate.edu.

Talin Mirzakhanian (BS, California State Polytechnic University) graduates with thesis "Multi-rate Runge-Kutta-Chebyshev Time Stepping for Parabolic Equations on adaptively refined meshes" (May, 2017)

Melody Shih (Master's student, Columbia Univ.) is working with D. Calhoun and Kyle Mandli (Columbia Univ.) to port GeoClaw, a widely used application for modeling tsunamis, storm surges and debris flows, to ForestClaw. Check back for progress!

Back to the top

Where can I get ForestClaw?

Forest Claw can be cloned/downloaded from GitHub, at http://www.github.com/ForestClaw/forestclaw. Please send us any feedback.

Back to the top

Developers

Lead Developers

Carsten Burstedde (University of Bonn, Germany)
Donna Calhoun (Boise State University, USA)

Developers

Melody Shih, (Columbia University)
Kyle Mandli, (Columbia University
Scott Aiton, Boise State University

Back to the top

Acknowledgments

Donna Calhoun would like to acknowledge the support of the Isaac Newton Institute and the program Adaptive Multiscale Numerics for the Ocean and Atmosphere, where much of the initial work in developing ForestClaw was done, and the National Science Foundation (NSF).

Gallery



Tracer transport


Gas dynamics

Last modified: Thu Mar 16 06:55:34 MDT 2017