Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make horizon finder faster for GPUs #6200

Open
geoffrey4444 opened this issue Aug 7, 2024 · 1 comment
Open

Make horizon finder faster for GPUs #6200

geoffrey4444 opened this issue Aug 7, 2024 · 1 comment

Comments

@geoffrey4444
Copy link
Contributor

geoffrey4444 commented Aug 7, 2024

Geoffrey's raw notes from the discussion.

08/07/24

Horizon finding chat at SXS Con 2024

Harald whiteboard talk

  • Nils is worried about horizon finding for GPUs...
  • Harald's idea:
    $dh/d\lambda = -f \Theta$
    Decompose into $Y^{\ell m}$.
    Shape map and control systems start out with a sphere

Shape map + control systems

Worldtube method: position evolving based on self force effects: $\ddot{\mathbf{x}}p = \frac{1}{m} \mathbf{F}{GSF}$. (GSF=gravitational self force)
Implemented: he chose the charge at the center of the sphere... charge is at the center. Mapped with control system with rotation and scaling to charge moving in inertial coordinates.

Niko has removed the control system, implemented ODEs for the mapping parameters

$$\left(\begin{matrix}\ddot{\phi} \\\\ \ddot{a}\end{matrix}\right) = f(\mathbf{F}_{GSF}).$$

Idea: define approximate position of horizon as grid coordinate sphere, radius $R$. Map via $r_{\ell m}$ to the inertial shape of the horizon. Idea is to determine

$$\dot{r}_{\ell m} = - (f\theta)_{\ell m}$$

from flow equation of apparent horizon. If this converges quickly enough, using real time instead of artificial time should still keep the surface tracking AH, at least approximately... you can put an extra timescale in, to make the tracking more accurate... add a constant to RHS $\ref{eq:blah}$

$$ \dot{r}_{\ell m} = - (f \theta)^{\ell m} $$

... no separate AH finders anymore for control systems... initial guess for approximate horizon finder...

Interpolation always happens at the same grid points.

Nils: I'm not worried so much about interpolation; 1 interpolation/timestep is fine

![[Pasted image 20240807162235.png]]
Mark: is this too restrictive, having horizon and excision surface both coordinate spheres?
You still have size control

Mark: in flow equation, there's also a 00 equation... because then you can't change $R$ ... Harald: yes you can

Nils: for gpus, we need to find the horizon faster...that's it

H.: can you do punctures in GH?
Nils Deppe: You must rederive if system is still hyperbolic, and there might be an issue with punctures being in the domain with spectral methods
Do you have to to the conformal decomposition? Wolfgang Tichy+ have tried GH conformal decomposition to do punctures, and their code is currently blowing up, not sure if it's at the DG FD boundaries because their DG+FD scheme is unstable, or if the equations just can't be solved... now implementing in CarpetX to see if it's the equations or their numerical scheme

Mark: spin 0.99 on both holes aligned, equal mass, average of 5-6 iterations on AhA / AhB every time in SpEC... for q=4, spin=0, at most 2 iterations

Larry: current tolerance on the AH finder?
Number of iterations in AH finder is a discrete number...

Nils remarks

We need...can we get these easier?

  • Safe but faster $\beta$ (adaptive?) Larry
  • Better initial guess (higher order and/or AH evolution): Mark
  • How long does AH finder take? Larry: how long to interpolate, and how long does FastFlow take? Kyle/Geoffrey/Mark
    • Which part is slow?
    • How many iterations? Geoffrey & Kyle (SpECTRE), Mark (SpEC)
  • Mesh is 3/2 LAH. You can make that L+2 or +4 instead of 3/2 LAH? : Mark
  • RK2 instead of forward Euler in $\lambda$?
  • Other horizon finder algorithms?

Nils: Stateless size!! this is going to limit our convergence at some point

Harald: I worry that we are tying spectre to one specific way of doing things; reduces flexibility... ties AH finder and control systems together on a deep level

Harald: Different modes converge at different rates; $\beta$ controls how much you overrelax the normals.

Discussion

Larry: another speedup that helps both: current algorithm has L representation and L to which you interpolate in order to solve the problem... currently Lmesh is 3/2 LAH. You can make that L+2 or +4 instead of 3/2 LAH?
Harald: I remember having lots of AH failures with little difference in resolution

The issue for GPU code?

Now need to find the horizons and update the control systems...
I need to do horizon find faster than GPU takes to take a timestep...

Nils Deppe: this might be hard, but I think it would solve this

Larry: multiple AH finding methods; FastFlow was added to SpEC in 2006, because it beat any finder known then...but it's limited in that it's adapted to spherically shaped horizons... and the expansion sign has to change correctly... there are other algorithms that if you have infinite speed on GPUs that might work better... usually they are global, return a scalar function and solve an elliptic equation for the function; this function's zeros are the horizon... after solving it everywhere, you have to find the zeros. The challenge was the elliptic solve.

GL: Elliptic solves on GPUs? Yes, that's machine learning
Parabolic solutions? CFL condition much worse...
First thing is how slow is the AH find...

Larry: if you choose $\beta$ wrong, it diverges, so if you catch this you can just lower $\beta$ and try again

Larry: This won't work for AhC

Overall options:

  • Make current scheme faster and/or have something that only requires the expensive iterative procedure less often
  • Evolving horizon in time
@geoffrey4444
Copy link
Contributor Author

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant