日拱一卒
0%

Apache HC Fluent API Hangs When Got Some 404

Posted on

jstack

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
"pool-1-thread-1" #19 prio=5 os_prio=0 cpu=4292.67ms elapsed=6105.93s tid=0x00007f7bb1a83670 nid=0x1f63b2 waiting on condition  [0x00007f7b748fa000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.1/Native Method)
	- parking to wait for  <0x00000000c7541680> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.1/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.1/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.1/ForkJoinPool.java:3463)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.1/ForkJoinPool.java:3434)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.1/AbstractQueuedSynchronizer.java:1623)
	at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:393)
	at org.apache.http.pool.AbstractConnPool.access$300(AbstractConnPool.java:70)
	at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:253)
	- locked <0x00000000c89eb980> (a org.apache.http.pool.AbstractConnPool$2)
	at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:198)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:306)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
	at org.apache.http.client.fluent.Request.execute(Request.java:177)
	at xyz.csongyu.smartinvesttask.configuration.FundOpenFundInfoEmTaskConfiguration.lambda$unitNetAssetValueRunner$1(FundOpenFundInfoEmTaskConfiguration.java:85)
	at xyz.csongyu.smartinvesttask.configuration.FundOpenFundInfoEmTaskConfiguration$$Lambda$567/0x0000000800f074c0.get(Unknown Source)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(java.base@17.0.1/CompletableFuture.java:1768)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

Code Example

csongyu/apache-hc-fluent-api

1
2
3
4
5
6
7
8
9
10
11
12
final List<Integer> indexes = Stream.iterate(0, item -> item + 1).limit(101).collect(Collectors.toList());  
final ExecutorService executorService = Executors.newFixedThreadPool(10);
CompletableFuture.allOf(indexes.stream().map(index -> CompletableFuture.supplyAsync(() -> {
    try {
        Request.Get("http://127.0.0.1:8088/index/" + index).connectTimeout(1_000).socketTimeout(5_000)
            .addHeader("Accept", "application/json").execute()
            .saveContent(directory.resolve(index + ".json").toFile());
    } catch (final IOException e) {
        return false;
    }
    return Files.exists(Paths.get(index + ".json"));
}, executorService)).toArray(CompletableFuture[]::new)).orTimeout(30, TimeUnit.SECONDS).join();

Default Pooling Connection Manager

Class Executor

A PoolingHttpClientConnectionManager with maximum 100 connections per route and a total maximum of 200 connections is used internally.

How to Fix

execute

Please Note that response content must be processed or discarded using Response.discardContent(), otherwise the connection used for the request might not be released to the pool.

discardContent

Discards response content and deallocates all resources associated with it.

1
2
3
4
5
6
7
8
9
10
11
12
13
Response response = null;
try {
    response = Request.Get("http://127.0.0.1:8088/index/" + index).connectTimeout(1_000)
        .socketTimeout(5_000).addHeader("Accept", "application/json").execute();
    response.saveContent(directory.resolve(index + ".json").toFile());
} catch (final IOException e) {
    return false;
} finally {
    if (response != null) {
        // important
        response.discardContent();
    }
}